Perhaps Vader recognizes Han as the one who sent him flying in ANH?
This is made explicit in the radio adaptation, and makes you wonder why similar dialogue wasn’t implemented in the movie. You kind of forget that Vader would probably be holding an enormous grudge against Captain Solo.