I wanted to determine, with de-noising the picture as an example, whether working in RGB (if you're considering it) is worthwhile ... practically speaking. I wouldn't to steer you into more of work if the lesser is just fine. :)

MVDeNoise() is a temporal de-noising script for Avisynth that overlaps a specified number of frames to make the "noise", which is random by any practical measure, cancel itself out. To take into account any picture movement, to minimize temporal smearing, it includes picture tracking. Pretty cool.

Because of HD picture size, I worked up a template to visualize cropped strips of the picture that would be displayed at full pixel size, including intermediate processing and final result -- 5 sections: "raw", "R", "G", "B", & "processed" -- all within a DVD-sized picture:

The first run is the raw image, it's RGB components (to show where the degree of noise resides), and the processed image. For this, MVDeNoise() was applied to the raw image only for the processed result:

The second run is the raw image, it's RGB components (MVDeNoise() was individually applied only to these channels), and the processed result:

From the first set, it is obvious the noise is greatest in Blue and least in Red. Therefore, in the second set, I had MVDeNoise() process +/- 1 frames in Red, +/-2 frames in Green, and +/-4 frames in Blue. This way, the settings produced approximately an equally de-noised result in each of the channels.

Comparing the two approaches, at this point in the restoration, doesn't show allot of difference ... at least in a still picture. Of course, the "de-noiser" in your software may show a greater difference depending on it's type of de-noising and how it implements it. Just FYI.