Sorry, but no. There have been hundreds of changes done between the originals and the 97 SEs.
The 2004 DVD is basically the 97SE with corrected colours
That’s just not true. The only people who ever say that are people who swear by the original cuts and don’t watch the special editions. I saw the faces release first, that was my intro to star wars when I was a kid, then I watched the special edition growing up, and then I had the DVDs when I was a teenager. Those are just what I had available because they didn’t rerelease the originals. The DVD version is completely different because the changes you describe warp the foundation the films. Adding a cgi dewback is just not the same as adding a Hayden Christensen.
Well it’s difficult to say if you’re judging it based on the perceived intensity of the changes, or the raw number. As far as number is concerned, I think the 97 will come in first for the change count (especially for ANH). In 1997, Lucas did say Jedi needed the fewest changes as it was in better shape in his mind. And they’ve changed it quite a bit since then. Still, the tons of little changes in optical effects and such are probably most numerous from original to 97.
It sounds more like you’re discussing severity of the changes, which is a pretty subjective thing. But, indeed, there wasn’t the (fully formed) corrupting influence of the PT in 1997. So for any of us PT critics, the 2004-2011 changes will carry a little more weight in how awful they can be. Some things in 1997 were probably Lucas’s original vision. Other things are surely bullshit. In another sense, the 2004 version could maybe be considered “most changed” since they butchered essentially every frame in color correction. And if light sabers are among your favorite things in the Star Wars universe, the visual effect is utterly trashed starting in 2004 (and not much better in 2011). So, yeah. It’s subjective.