I've written about this before, but I'll summarize. I was seven when Jedi came out and I loved it. So did all my friends. But I remember my mom complaining about some things after we saw it, like Luke & Leia being siblings.
I noticed that older kids (those that were teens when Jedi came out) did not hold the movie in as high regard. Usually the Ewoks were the divisive element for them.
As my friends and I got older and our tastes developed, we too began to notice how much better made ANH & ESB were. All this was pre-WWW, BTW. So by default, ROTJ became the worst of the trilogy. Along the way, I suppose, people might have misconstrued "worst of the trilogy" into "lousy movie."
Of course, now it's become the 3rd best of the Hexalogy.