From my experience with women being considered a handsome man is more about your choices in how you present yourself than anything else. Being healthy, well groomed, knowing whether you look better with or without facial hair, hairstyle, being at a healthy weight etc. Mostly things you can control.
I’m anemic, ungroomed, underweight and not interested in controlling any of it. You are right though.