There are 2 real problems:
1) Some workers are not balanced for 2x, 4x speed
2) Strong teams take all good workers. This creates a huge imbalance.
I always play in strong teams, but I still consider it unfair.
Many players and teams complain about this.
What I suggest (better version):
A) Each team can only hire 1 worker at a time
B) Workers are divided into groups (groups according to value). Each team can only hire one worker per group
C) Workers are divided into groups (groups according to value). Each team can hire only one worker from the most valuable group