I will agree that there is one big difference between a male being harassed and a female being harassed. Most men are bigger and stronger than women, that is just the biology works. So when a man gets harassed, he can defend himself because he is stronger, most of the time. But when a woman gets harassed, most of the time she can not.
I suppose they could, but how would that reflect on them? You may be able to punch a male harasser (even that would look bad, as Terry Crews said); but hitting a woman is a huge no-go, coincidentally for the exact same reason that you say men should be able to defend themselves.