DISCLAIMER: I know very little about what I'm about to say :).

I was thinking about this and I'm wondering if better results could be had by cropping the frames to 2 wide rows of horizontal or vertical lines of pixels and then running Colourlike on each row and then pasting them back together to make the entire image.

It should keep the filter from applying values to the entire image and limit it to a certain area. I'd imagine both images would have to be aligned fairly well.

Maybe something like the line below could be used instead of Colourlike.

ScriptClip(clip1,"ColorYUV(off_y(averageluma(clip2)-averageluma(clip1)),off_u(averagechromau(clip2)-averagechromau(clip1)),off_v(averagechromav(clip2)-averagechromav(clip1)))")

I'm not sure if it's any faster or more accurate but I've had some good results with it before.

Edit: After playing around with this idea for a little bit I now realize it was probably a stupid idea.