That would be very possible if it were a simple zoom or panning shot, unfortunately it’s much more complicated so significant castle damage would probably have to be modeled in 3d and tracked onto the existing shot. Needless to say it’s a bit beyond my capabilities.
A thought I had was potentially just tracking some cracked/crumbly textures onto it, so at least it doesn’t look so pristine.