Rivalry only works whilst you hold the majority. In this situation you are forcing the free-hauler to break your majorities. The rival setting is almost always counterproductive.
I know players want to be able to force everyone to do what they want (I've felt the same many times), but you shouldn't have that ability. Unless they have agreed to work with you (by joining your association, asking for your friendly or some other deal) then they owe you nothing. If you don't have enough active players to combat them then tough. You were going to lose anyway, at least now you have someone to blame.