Darth Lucas said:

For ESB and ROTJ I only plan on having 1997 because the differences between those and 2004 are so minuscule.

I think the difference of Hayden in ROTJ 2004 is something that most of us here would consider "enormous". Certainly on par with changing Jabba's model in ANH.

Empire's changes are smaller in general, but I think the 1997->2004 changes are certainly noticeable: New Emperor, new Boba voice, removal of scream.

But also, this is a Blu-ray set. The 2004 editions can be included at 1080p and fairly high quality. The 1997 editions will never look as good as the rest of your set.

If your goal isn't to release every version (and it doesn't seem to be), then don't sweat it. But you might just want to be consistent and use a set of 1997, or 2004, or 2011 rather than mix it up for each film.