Help Wanted: Great White (1998) - Visual Cleanup [D2 Master/8bit]

So, two years ago, I was given a license to distribute this '90s DTV shark film entitled: GREAT WHITE. I released the DVD one year ago, but am looking to fix up the 8bit YUV422 master file. It was shot on BetaCamSP at 60i and during the editing process, it was deinterlaced to 30p. It was then mastered onto both D2 and BetaCamSP. I was given the D2 master.

Off the bat, I’m dealing with stair-stepping, which I know cannot be really dealt with, but the biggest issue is the color banding artifacts present in the video. Examples below (Click Them):

I have the uncompressed YUV422 8bit file on my hard drive, so I do have an advantage for quality, but are there any tools that I can use to help make this video look any better and remove the banding?

JAWS

http://avisynth.nl/index.php/GradFun2db

Yeah, "gradfun2dbmod" is what I use as an antibanding filter.

This was my result.. didn't do anything, just added those three stripes. Am I doing something wrong? Rendered out Sony YUV Codec from Vegas and then brought it into VirtualDubMod.

Tried both of those, nothing's working so far. Any other options?

Did you try chubbyrain2?

RU.08 said:

Did you try chubbyrain2?

 Tried. Won't let me open it in VdubMod. I keep getting: AviSynth Open Failure: Evaluate System Exception - Integer Divide by Zero. (ChubbyRain2.avsi, line 18)(L:\deband.avs, line 2).

Being that the video was deinterlaced from the original BetaCamSP tapes, would that make this process impossible?

No interlacing shouldn't matter. Did you make sure to put converttoyv12() in your script? Many filters including chubbyrain2 and gradfun2db/mod are only compatible with YV12 and don't work even with YUV (which is the interleaved equivalent and often the colourspace supplied to Avisynth). If you want to keep 100% quality (i.e. the full luma quality) you could try doing this:

pointresize(width(),height()*2)
converttoyv12()
chubbyrain2() #or gradfun2dbmod()
converttoyv16() #or converttoyuy2()
pointresize(width(),height()/2)

# (edited to fix the linebreaks)

Also, I suggest using AvsPmod instead of VDubMod which is obsolete now. For the credits if they're all black-and-white you could just convert that section to greyscale using greyscale().

RU.08 said:

No interlacing shouldn't matter. Did you make sure to put converttoyv12() in your script? Many filters including chubbyrain2 and gradfun2db/mod are only compatible with YV12 and don't work even with YUV (which is the interleaved equivalent and often the colourspace supplied to Avisynth). If you want to keep 100% quality (i.e. the full luma quality) you could try doing this:

pointresize(width(),height()*2)
converttoyv12()chubbyrain2() #or gradfun2dbmod()
converttoyv16() #or converttoyuy2()pointresize(width(),height()/2)

Also, I suggest using AvsPmod instead of VDubMod which is obsolete now. For the credits if they're all black-and-wite you could just convert that section to greyscale using greyscale().

Finally got some decent results. Removed a good portion from what I can see. The next issue would be ridding the video of the haloing & more crosstalk. I understand that I'd lose detail, but I'm trying to make a better master. Any suggestions?

Here's an example of what I believe to be crosstalk?:

Here's a sample (YV12 AVI) of the clips with the most chroma crosstalk, can anyone give a basic script an attempt and see if it's salvageable?

https://drive.google.com/file/d/0B1-W6ByLSZVKN19sQ29ReVBBdXM/view?usp=sharing

Okay, so the de-rainbowing setting I used was chubbyrain2(th=3,sft=20) - it seemed to get rid of all the rainbowing in that clip, but of course you'd have to look over the whole file to find out if you need to use even more aggressive numbers than that (and I'd suggest to tune it so you don't use more aggressive numbers than you need since it'll make the chroma look flatter). I don't think you're going to be able to do much about that chroma bleeding - you could have the affected scenes professionally re-coloured, but I don't think there'll be much you can do about it in Avisynth. On the plus side there's not much bleeding in the scene you provided.

Here's a quick script, see what you think. It takes care of the haloing and aliasing (from the deinterlacing) as best as I can. You can't go too agressive on the dehaloing or you'll make a visible haze around the hard edges. You can use inputtype=2 in QTGMC if you want however the anti-alising will work better with the interlaced mode on and there should be a negligible difference in quality (it's negligible in the sample anyway).

#NOTE: Much better script in next post!

orig=avisource("great_white_clip.avi").killaudio()
orig.ChubbyRain2(th=3,sft=20)
QTGMC(Preset="Placebo",TR2=0,NoiseProcess=2,NoiseRestore=0,GrainRestore=0.5).selecteven()
centre=crop(10,0,-10,-2)
centre=centre.BlindDeHalo3(rx=3,ry=2.5,strength=120,sharpness=0.5,tweaker=0)
centre=centre.LSFmod(defaults="slow",strength=120)
stackhorizontal(crop(0,0,10,0), \
    stackvertical(centre,crop(10,478,-10,0)), \
    crop(710,0,0,0))

# Use this line to compare to original
# stackhorizontal(orig,last)

Good luck!

Before:

After:

Here's an improved script with no need to do any sharpening now. It throws away the existing interpolated lines (the even lines) and starts again, quality is improved over the version I posted above. If you wanted you could put LSFmod it back in with a conservative value for strength in the range of 30-60 or so.

The GrainRestore setting improves the sharpness, so if you boost it from 0.7 to say 1.0 it will sharpen the image also. Personally I'd leave it on about 0.7 and use LSFmod with a setting of about 40 to bring the sharpness up a bit if that's what you want - but I don't think it needs to be quite as sharp as the source (which is due in part to the sharpness of the aliasing).

orig=avisource("great_white_clip.avi").killaudio()
orig.ChubbyRain2(th=7,sft=15)
converttoyuy2()
separatefields()
assumeframebased()
selectodd()
nnedi3(dh=true)
stackvertical(crop(0,1,0,0),orig.converttoyuy2().crop(0,479,0,0))
converttoyv12()
QTGMC(Preset="Placebo",TR2=0,NoiseProcess=2,NoiseRestore=0,GrainRestore=0.7).selecteven()
centre=crop(10,0,-10,-2)
centre=centre.BlindDeHalo3(rx=3,ry=2.5,strength=150,sharpness=0.5,tweaker=0)
stackhorizontal(crop(0,0,10,0), \
    stackvertical(centre,crop(10,478,-10,0)), \
    crop(710,0,0,0))
#stackhorizontal(orig,last)

Quick notes: Retaining the video grain (GrainRestore setting in QTGMC) won't do anything for you when authoring for DVD since it'll all be blurred out by the MPEG2 compression, but it will look nice if you ever decide to release it as MPEG4 AVC. You don't want the video looking "processed".

Screenshot of this script...

Before:

After:

By the way, I posted this in another thread, if you had the desire to make it 60i you could just add this to the end of my script and you'll get a full proper interlaced version (a very close approximation to what the original 60i video would have looked like):

even=last.converttoyuy2().separatefields().selecteven()
threads=10
super_params="{pel:2,gpu:0}"
analyse_params="""{block:{w:8,h:8,overlap:3},
    main:{search:{coarse:{distance:-10}}},
    refine:[{thsad:200}]}"""
smoothfps_params="{algo:21}"
super = SVSuper(super_params)
vectors = SVAnalyse(super, analyse_params)
SVSmoothFps(super, vectors, smoothfps_params,url="www.svp-team.com",mt=threads)
odd=converttoyuy2().selectodd().separatefields().selectodd()
interleave(even,odd)
weave()

Probably not a good idea for release, but not a bad one if you want to watch it on a CRT tellie.