I can't remember where I heard it, but Ben Burtt said that most people were not impressed and disappointed with Empire. Over the years, though, most people fell in love with it and said it's the best out of the three. Maybe it wasn't loved at first because it wasn't as action oriented as Star Wars?
I love it because of the emotion with Yoda, Vader's reveal to Luke, the scenes in the astroid field and on Hoth, Boba Fett, and the towards the end it was SO VERY suspenseful. Overall it captured the most emotion, I think.