That's pretty much what the Doctor Who restoration team did with some of the early 1970's episodes that only survive as black and 16mm prints. They used the color information from vintage off air video recordings provided by a fan.
http://www.purpleville.pwp.blueyonder.co.uk/rtwebsite/colouris.htm