Monogamy has downsides too. Like sexless marriages.

But in a polygamous society men on average would have much less sex, as the most attractive men reap all of the spoils.