Thanks for the sample Harmy, I'll do some tests later or tomorrow.

Well, RGB => YUV420 is a lossy conversion (rounding errors etc.). It does however consider the fact that the human eye is much more sensitive to brightness than it is to color and thereby achieves a much better compressibility. So I think it's a combination of both. But even if that much color information is thrown away, you should not see much of a difference. The color shift that we see here should not happen I think.

Also, you have to remember that your source is not RGB anyway. Most of it originates from Bluray, HDTV or DVD, which already use YUV420.