It's a perception thing--when someone is really attractive human jealousy tends to make us think less of them in other areas to lessen the blows to our ego, or, if the opposite sex, to make them seem less imposing.
On the otherhand, there are some jobs where 'sexy attire' probably wouldn't be appropriate. *coughsundayschoolteachercough*