This glitching in the highlight areas has always been a problem, and is a big reason I never used this method in in my project. The artifacts look similar to artifacts generated by Photoshop’s Hue/Saturation tool, where shifting the hue of a particular color runs into problems when it encounters high luminosity or low saturation areas. These areas are shifted to the same extent as saturated midtones, leading to blocks of darker or lighter colors in almost colorless walls or skies. I don’t know if this is what the problem is in this case, but perhaps the program would benefit from an algorithm which takes this into account and shifts color less and less as the luminosity approaches 255 or 0 and the saturation approaches 0.