I held jealously onto my primacy bias until just last year. I watched the GOUTs in order, to see how they looked on my new-ish 37" LCD (terrible).
I realized that ESB was just way more entertaining from moment to moment than SW. (Yes, and that ROTJ kinda sucks.)
SW is ultimately a series of ingenious set pieces. (Droids; Jawas; Cantina; Compactor; Deactivation of Tractor Beam; Canon-cockpit Tie dogfight; etc.) As exciting as each set piece is, it's all slightly jarring continuity-wise.
ESB, on the other hand, flows and sweeps with cohesive and compelling perfection. The performances are genuine, the structure and pacing are quickquickquick but natural. It's just irresistible and engrossing on a level that SW just had not reached.
I just think it requires appreciation from a more sophisticated perspective than the others, so people typically embrace it with age and wisdom.