That's great. Actually, it's a many to many mapping, not a one to many mapping. You should save the rotations, and the mapping in each color space, and then apply each of the rotations and mappings to the new image, transform the individually mapped images back to your original color space, and average all the predicted images.
Update: there's a bug in my code, so mapping multiple color spaces for different images/frames is not yet possible.