Yeah, I agree.

But men did actually, physically, build the world with their hands.

They couldn’t have done so without women. But saying “men built the world” isn’t exactly wrong.