As far as I can tell, changing the rendering quality on the project tab only effects the quality of the preview window during the render.

I never said the quality difference would be night and day. It depends on what you are encoding. Furthermore, the preview window is showing you what the video will look like at that level of quality. The problem is, you don't know what differences to look for, so the differences aren't apparent to you.

Find a scene with fast motion, and view the frame under GOOD quality, and then PREVIEW quality. GOOD quality shows the differences between the fields, sometimes resulting in a double-image (motion between the fields). The PREVIEW quality isn't as precise, so it only shows one of the fields. at 30 fps, this isn't a big deal, but when I wanted to slow down the video, the difference between fields was obvious. Therefore, I rendered this section out under PREVIEW quality, ran it through my SlowMotion program, and put the new segment back in the timeline. Later, I rendered the whole video, along with this new segment, in GOOD quality.

Video under GOOD quality:

Video under PREVIEW quality:

As you can see, there are much cleaner lines in high motion videos.