It’s not the term itself, it’s how it’s used. And it’s sexist because no one ever accuses male characters of being a Mary Sue, or a Gary Stu, or Harry Flu, or whatever.
Wesley Crusher (a man… sort of) is quite possibly the most referenced Mary Sue character in the history of fiction. He is even mentioned in the Wiki page!
Well if it’s on Wiki it must be true.
I’ve literally never heard a single male character called this other than what you just said, and I’m not the only one.
And sexism takes many forms. You don’t have to be physically assaulted to be a victim.