With polygyny the most fit are mating far more than the avg fit people are, so that compensates for them being a minority. Kind of like how the richest 5% of the world has 95% of the wealth or w/e. The fittest 5% would do 95% of the mating.
Regardless of our fitness, monogamy makes us all pretty much have the same amount of offspring, for better or worse.

*This is to do with male mating habits primarily.
**Not advocating it, just raising the point