Simply put, Fett is one of the only characters with the balls to talk straight with Vader and not cower. The guy had only four lines in the OT, all of them in Empire. Two of these lines were basically back-talking to Vader.

"He's no good to me dead."
"What if he dies? He's worth a lot to me."

These are in essence the same friggin line, but the point is, he's willing to say what he thinks to the most ruthless badass in the galaxy -- straigh to his face (er, mask). That make him a serious badass in his own rite.

Tarkin was this way too, and it made him almost as cool. Barking orders at Vader. But clearly they had a 20-year relationship in place.

Motti and Ozzel were less fortunate.