Without any further context, that's what I would go with. I assume the engine somehow manages or (even) generates rules, so the "rule" here really belongs to and should be connected to the "engine". Also compare "sökmotor", grafikmotor etc.
Or if you want to avoid compound words (I favor avoiding compounds as they tend to easily become awkward or unwieldy), i'd go with "motor för verksamhetens regler".
(I generally also prefer "verksamhet" before "affär" as "verksamhet" is broader for any business activity, while "affär" is more limited to purely financial transactions.)