@ Laserdisc Master
Thanks a lot! I cut a few bars, because CatBus explained to me that the first 8 and last 8 pixels are not supposed to be in the frame. This leads to a 704x480 frame, displayed at 640x480, with an aspect ratio of 4:3. When upscaling 3x you get 1920x1080. So the only way to get the correct aspect ratio is to scale the 704x274 frame to 720p or 1080p. The reason I use 1920x816 is because it corresponds to an aspect ratio of 2.35:1.
For more info: http://en.wikipedia.org/wiki/Nominal_analogue_blanking