There is nothing sexist about the term itself, but it has been used in a sexist way so many times that it has this connotation. I think we all know other words that aren’t used anymore for similar reasons.
I don’t find “Mary Sue” offensive, I just think it’s a too broad term to accurately convey one’s opinion. Everyone has their own version of what exactly a Mary Sue is and what isn’t (or they don’t know the term and have to rely on someone else’s definition) and everytime the term is used, these different interpretations lead to arguments that have nothing to do with the character itself.
I think it’s best to avoid the term and instead directly say which aspects of a Mary Sue you think apply to a certain character. This avoids the confusion and the need to have these arguments again and again.