My dislike of one and apathy towards the other comes from the films internal consistency.

Imagine watching ESB for the first time. It does not matter which version you watch, all you know is that this is a brand new character and it’s consistent in that film. When you switch over to return of the Jedi and the actor changes (if you watch the theatrical version) then you just assume stuff happened behind the scenes and this is now the Emperor.

The problem with ROTJ is that Hayden at the end breaks the films internal consistency. I don’t know over any other film (besides that movie for Heath Leger after he died) that had multiple actors embody the same character. If you’re a first time viewer of ROTJ and it’s the first Star Wars movie you’ve seen ever then it’ll be really odd to see a brand new, never before seen, actor portraying… well you can’t even be sure if you have no prior knowledge to bass it off of. That is why the replacement of Sebastian Shaw is so much more reviled then the replacement of the Emperor.