The Blu-ray mix isn't really all that different from the 2004 DVD mix overall, AFAICT.

Badness depends on your reference. So many of us have seen the films so many times that we have at least one of the pre-97 mixes permanently etched into our brains, so that's our reference whether we like it or not. Maybe if you judge the Blu-rays without that mental reference, they sound okay, but for me it's impossible, because I also know how they could/should have sounded.

So here's what I remember not liking about the Blu-ray audio. Please keep in mind I'm not really an audio guy, so my terminology or explanations might be completely wrong. They had a weird combination of muddiness and harshness throughout, which I took to mean overzealous use of noise reduction (muddiness), combined with a hamfisted attempt to bring the reduced details back out (harshness). Usually this was constant, so your ears could get used to it, I suppose, but occasionally they must have done some major adjustments, and the sound suddenly sounded more broken than normal. I remember that happening with Vader dialogue occasionally.

Then there's the score, which was pushed way into the background in a few scenes--yeah, it's not like the score isn't the best part of the soundtrack, people--let's just minimize that John Williams garbage so people can hear our crap new sound effects for our crap new CGI. Bleah. Bad choices all around, even if you accept the bad choice of the Special Editions to begin with.

EDIT: hairy_hen may be your best bet for more detail. A lot of people who don't like the 2011 mix may have only heard it once quite a while ago, or be actively trying to forget it. It's not something I hope to re-live.