So after much testing, I've decided to drop in the asharp filter to bring up some of the detail lost in the clean up.
I'm usually opposed to edge enhancement, but in this case it looks quite good.

Now I have a tech question for the crowd.
The restore24 filter is VERY SLOOOOOOW.

To do a 5 pass encoding will probably take me something in the neighborhood of 30 hours (give or take)(estimate is based on an OPV encoding of 5% of the film).

Since the filter is the limiting factor is there anything wrong with me instead taking the avisynth output and encoding to a lossless format (HuffYUV or Lagarith) and then doing a mulipass CCE encode from the lossless version? I have to figure that'll cut the encoding time in half or better.

Is there anything wrong with that idea?