The simplest explanation is the OT/PT divide. Many fans hate the PT and would rather pretend it not exist while watching the OT. Even though Ian looks as he does in ROTS, it can be ignored simply because he’s the right actor for the OT as well. But Hayden, for a lot of people he has no business being in the OT because he wasn’t in to begin with (not to mention that for many he represents the worst the PT has to offer).
I take a more reasoned approach. First of all, I almost exclusively watch the OOT, so who cares. But when we’re talking about an SE, one that is supposed to make the saga more coherent as a whole, I tend to think along these lines…
Every time these discussions come up the most simple answer is always ignored; Hayden was put in because the audience can actually identify with Hayden’s Anakin. Shaw’s Anakin is never once seen at any point until the final minutes of Return of the Jedi and Hayden’s Anakin has two entire movies and almost five hours of screen time dedicated to him. If you put Return of the Jedi in the context of the six film saga then Shaw’s Anakin is visually out of place, it’s really that simple.
In the context of the whole saga, Hayden is Anakin. Sure, you can understand that Shaw is supposed to be Anakin, but the fact of the matter is he never once looked like that in any of these films. Not to mention, continuity-wise, Shaw is far too old to be playing Anakin (he still works as the unmasked Vader because his injuries, the suit, and the dark side have made him more decrepit). You can say that it doesn’t make sense that the ghost would look like he did decades before he died, but I’d argue it makes as much sense as an imagined fantasy version of older him.
Now, the other thing people complain about is that the Hayden footage was a costume test, and it doesn’t fit. This is true, and it’s a little janky, but it’s not that bad. Here’s the thing though. The people saying Shaw’s acting is not being respected? That’s silly. Shaw’s acting is intact, in his scene with Hamill. Shaw’s ‘acting’ in the ghost scene was literally just George telling him to smile at the camera. That’s it. He had no idea the context, what was going on, or why he was smiling. So I’d say that’s only marginally better than a costume test.
Ultimately what it comes down to for me is that Anakin/Vader shouldn’t end up with a ghost at all at the end of the movie, so both versions are stupid to me, personally.