I think what hurt ROTJ most was the, "knock it out quick, it'll do" attitude of Lucas which percolates through the whole piece.
The story is practically a cut an paste rehash of elements from the previous two stories.
The director was Medvedev to Lucas' Putin.
The sets were largely a collage of bits left over from the other films and photographed with very little sense of cinema (compare Jerjerrod's control room to Tarkin's in ANH, there are 1970's Doctor Who sets that look practically the same, no wonder most of the footage ended up on the cutting room floor).
In terms of characterisation ESB was a big step up from ANH because it had to be.
The audience had already entered the world and fallen in love with the core characters so the next film couldn't just coast along on the same level.
Lucas had the good sense to get a director in who could not only handle/control actors but also listened when they had good ideas which gave them a sense of personal investment in their roles.
Mark pretty much kept that going in ROTJ but almost everyone else seemed to be in it to close that paragraph of their resume/CV.
This is why to me Jedi hurts more than the PT.
I can choose to ignore it the PT but if I want closure from ESB's storylines I have to sit through ROTJ so it becomes as much of a contractual obligation to myself as it was to the people who made it.