Shrinking a DVD9 to a DVD5 will definitely reduce the quality. If you're using something like DVDshrink, it will certainly hurt the video quality, because it is just transcoding the video, which is bad, but the quickest and easiest way to shrink a DVD9 to a DVD5. Try using something like DVDrebuilder (there's a free version and a paid version; the free version is pretty complete as is). It uses external encoders, which can become expensive, but I believe it comes with one built-in for starters. The difference between this and, say, dvdshrink, is that this actually re-encodes the video to fit on a DVD5, instead of just compressing it, so the quality can be dramatically better. Since I've used it, I've never looked back at DVD shrink for DVD9 conversions again. The downside is that encoding can take several hours, and setup can be a bit tricky if you're new at this, but there is a manual on the site, as well as plenty of online help.

Not sure why you couldn't burn to a DVD9 though, shouldn't be a problem. Could it be the burning software you're using?