It's best to manually fix the damaged frames and to do a little bit of manual dirt cleanup first because once you've done that, you'll have a "pristine" reference video that you can then try all sorts of video filters on. So lets say that you apply a filter on your manually cleaned reference video and it smooths over too much image detail - well, not to worry, because you can just start over again and try different settings or other filters on the reference file. The major plus of this is that you won't have to keep going back and having to fix those badly damaged frames every time you want to try using different video filters/settings. This will save you time and allow you to do more experimentation, which will hopefully get the video image to look as good as it possibly could.