If you’re going to be doing any reencoding, particularly for color alterations, it’s always going to be a better idea to convert it to a lossless (or near-lossless) codec first. H264 is… not that.
I thought CRF=0 was lossless for h264/x264. That’s what I’ve read everywhere on the web at least.