Sign In

GOUT Anti-aliasing

Author
Time
I wanted to make anamorphic versions of the GOUT and in the process I've been experimenting with some avisynth filters (EEDI2/limitedsharpenfaster) to remove the terrible "jaggies" without introducing too much side effects and loosing too much sharpness. Although the results are far from perfect I think the picture quality is a bit better. Here are some results (bottom pictures are filtered):
http://img526.imageshack.us/img526/290/1ata6.jpg
http://img526.imageshack.us/img526/6537/1byc9.jpg
http://img526.imageshack.us/img526/9143/2ado2.jpg
http://img526.imageshack.us/img526/6047/2bfl2.jpg
http://img526.imageshack.us/img526/3083/3anq5.jpg
http://img526.imageshack.us/img526/4664/3bgg4.jpg
http://img526.imageshack.us/img526/9286/4aei0.jpg
http://img526.imageshack.us/img526/8896/4bxu2.jpg
http://img526.imageshack.us/img526/9283/5afn1.jpg
http://img526.imageshack.us/img526/6121/5bet1.jpg
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
Are you upscaling to 720p?
(A thumbnail and a link would be kinder to those on slow connections... )

Good work; I was looking to try something along the same lines. Are you doing any kind of noise reduction/grain removal as well? What's your script?

Guidelines for post content and general behaviour: read announcement here

Max. allowable image sizes in signatures: reminder here

Author
Time
No, I'm not upscaling. I took screenshots of 720 x 480 material at 200% in virtualdubMPEG so the detail would be a bit more visible. Removegrain is used in limted sharpen. I only found out about EEDI2 today so I just used the standard settings. Something called "AAA" used before EEDI2 should give better result or so I read but I have no idea what .dll is required for AAA.

I resized before using limited sharpen but it should not be necessary as limited sharpen can be used to risize but I didn't look into that yet.

One of the bad side effects I noticed is that the stars in the star field are a bit less bright.

Sorry about the big pictures but if I used thumbnails a side to side comparison would be more difficult.

Here's the script I used:




LoadPlugin("DGDecode.dll")
MPEG2Source("[2]_VTS_03_1.m2v.pulldown.d2v")

loadplugin("eedi2.dll")
Loadplugin("mt_masktools.dll")
Loadplugin("warpsharp.dll")
Loadplugin("removegrain.dll")

Crop(0,102,0,-104,align=true)

EEDI2().TurnRight().EEDI2().TurnLeft()

Lanczos4resize(720,360)

function LimitedSharpenFaster( clip clp,
\ float "ss_x", float "ss_y",
\ int "dest_x", int "dest_y",
\ int "Smode" , int "strength", int "radius",
\ int "Lmode", bool "wide", int "overshoot", int "undershoot",
\ int "soft", int "edgemode", bool "special",
\ int "exborder" )
{
ox = clp.width
oy = clp.height

Smode = default( Smode, 3 )
ss_x = (Smode==4)
\ ? default( ss_x, 1.25)
\ : default( ss_x, 1.5 )
ss_y = (Smode==4)
\ ? default( ss_y, 1.25)
\ : default( ss_y, 1.5 )
dest_x = default( dest_x, ox )
dest_y = default( dest_y, oy )
strength = (Smode==1)
\ ? default( strength, 160 )
\ : default( strength, 100 )
strength = (Smode==2&&strength>100) ? 100 : strength
radius = default( radius, 2 )
Lmode = default( Lmode, 1 )
wide = default( wide, false )
overshoot = default( overshoot, 1)
undershoot= default( undershoot, overshoot)
softdec = default( soft, 0 )
soft = softdec!=-1 ? softdec : sqrt( (((ss_x+ss_y)/2.0-1.0)*100.0) ) * 10
soft = soft>100 ? 100 : soft
edgemode = default( edgemode, 0 )
special = default( special, false )
exborder = default( exborder, 0)
#radius = round( radius*(ss_x+ss_y)/2) # If it's you, Mug Funky - feel free to activate it again
xxs=round(ox*ss_x/8)*8
yys=round(oy*ss_y/8)*8
smx=exborder==0?dest_x:round(dest_x/Exborder/4)*4
smy=exborder==0?dest_y:round(dest_y/Exborder/4)*4

clp.isYV12() ? clp : clp.converttoyv12()

ss_x != 1.0 || ss_y != 1.0 ? last.lanczosresize(xxs,yys) : last
tmp = last

edge = mt_logic( tmp.mt_edge(thY1=0,thY2=255,"8 16 8 0 0 0 -8 -16 -8 4")
\ ,tmp.mt_edge(thY1=0,thY2=255,"8 0 -8 16 0 -16 8 0 -8 4")
\ ,"max") .mt_lut("x 128 / 0.86 ^ 255 *") #.levels(0,0.86,128,0,255,false)

tmpsoft = tmp.removegrain(11,-1)
dark_limit1 = tmp.mt_inpand()
bright_limit1 = tmp.mt_expand()
dark_limit = (wide==false) ? dark_limit1 : dark_limit1 .removegrain(20,-1).mt_inpand()
bright_limit = (wide==false) ? bright_limit1 : bright_limit1.removegrain(20,-1).mt_expand()
minmaxavg = special==false
\ ? mt_average(dark_limit1, bright_limit1)
\ : mt_merge(dark_limit,bright_limit,tmp.removegrain(11,-1),Y=3,U=-128,V=-128)

Str=string(float(strength)/100.0)
normsharp = Smode==1 ? unsharpmask(strength,radius,0)
\ : Smode==2 ? sharpen(float(strength)/100.0)
\ : Smode==3 ? mt_lutxy(tmp,minmaxavg,yexpr="x x y - "+Str+" * +")
\ : mt_lutxy(tmp,tmpsoft,"x y == x x x y - abs 16 / 1 2 / ^ 16 * "+Str+
\ " * x y - 2 ^ x y - 2 ^ "+Str+" 100 * 25 / + / * x y - x y - abs / * + ?")

OS = string(overshoot)
US = string(undershoot)
mt_lutxy( bright_limit, normsharp, yexpr="y x "+OS+" + < y x y x - "+OS+" - 1 2 / ^ + "+OS+" + ?")
mt_lutxy( dark_limit, last, yexpr="y x "+US+" - > y x x y - "+US+" - 1 2 / ^ - "+US+" - ?")

Lmode==1 ? mt_clamp(normsharp, bright_limit, dark_limit, overshoot, undershoot) : last

normal = last
zero = mt_clamp(normsharp, bright_limit, dark_limit, 0,0)

Lmode==3 ? mt_merge(normal,zero,edge.mt_inflate()) : normal

edgemode==0 ? last
\ : edgemode==1 ? mt_merge(tmp,last,edge.mt_inflate().mt_inflate().removegrain(11,-1),Y=3,U=1,V=1)
\ : mt_merge(last,tmp,edge.mt_inflate().mt_inflate().removegrain(11,-1),Y=3,U=1,V=1)

AMNT = string(soft)
AMNT2 = string(100-soft)
sharpdiff=mt_makediff(tmp,last)
sharpdiff2=mt_lutxy(sharpdiff,sharpdiff.removegrain(19,-1),
\ "x 128 - abs y 128 - abs > y "+AMNT+" * x "+AMNT2+" * + 100 / x ?")

soft==0 ? last : mt_makediff(tmp,sharpdiff2)

(ss_x != 1.0 || ss_y != 1.0)
\ || (dest_x != ox || dest_y != oy) ? lanczosresize(dest_x,dest_y) : last

ex=blankclip(last,width=smx,height=smy,color=$FFFFFF).addborders(2,2,2,2).coloryuv(levels="TV->PC")
\.blur(1.3).mt_inpand().blur(1.3).bicubicresize(dest_x,dest_y,1.0,.0)
tmp = clp.lanczosresize(dest_x,dest_y)

clp.isYV12() ? ( exborder==0 ? tmp.mergeluma(last)
\ : mt_merge(tmp,last,ex,Y=3,U=1,V=1) )
\ : ( exborder==0 ? tmp.mergeluma(last.converttoyuy2())
\ : tmp.mergeluma( mt_merge(tmp.converttoyv12(),last,ex,Y=3,U=1,V=1)
\ .converttoyuy2()) )

(edgemode!= -1) ? last : edge.lanczosresize(dest_x,dest_y).greyscale

return last
}

LimitedSharpenFaster()

addborders(0,60,0,60)
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
A suggestion on style - if you put the dll files into your avisynth plugins folder, they are loaded automatically. You can do a similar thing with functions; so you could create a text file containing LimitedSharpenFaster, save it in the plugins folder with the extension avsi and the function will be available from any script you create.

Simplified, your script then becomes:
MPEG2Source("[2]_VTS_03_1.m2v.pulldown.d2v")
Crop(0,102,0,-104,align=true)
EEDI2().TurnRight().EEDI2().TurnLeft()
LimitedSharpenFaster(dest_x=720, dest_y=360)
addborders(0,60,0,60)
... which is much easier to read.

(My quick calc found dest_y should be 365.3, say 366, so the addborders would be 57 each. The difference is probably not worth worrying about.)

"AAA" I believe was a function called Anime Anti Aliasing, now replaced by its author with a function called "SAA" (Simple Anti Aliasing). (Doom9 Thread). I haven't really looked into this myself yet so I'm still a bit fuzzy on specifics.

Guidelines for post content and general behaviour: read announcement here

Max. allowable image sizes in signatures: reminder here

Author
Time
Thanks for the suggestions. I know the dlls can be loaded automatically but when I started using avisynth that somehow didn't work. I'll try it again
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
Arnie, did you start with the PAL DVDs? If so, the jaggies are partly (if not entirely) due to the PAL DVDs being a simple upscale of the NTSC ones.

The jaggies do seem to be horizontal only, so perhaps you only need to EEDI2 once, and not turnleft/turnright. There's also a plugin from the same author called something like EEDIupsizer, which scales up by 2 in both directions in one go.

DE
Author
Time
No, I use the NTSC version. But in my opinion the jaggies are just as worse in the NTSC version.

I have no idea what this turnleft/turnright stuff does. What do you mean by using it once, only use turnleft or turnright or just use "IDDE2()" and leave out all the turn stuff? Aren't jaggies like this always horizontal only? They are caused by the pulldown scanlines right?
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
Your script does:

EEDI2().TurnRight().EEDI2().TurnLeft()

Which doubles the height, rotates the image 90 degrees, doubles the height (width) again, then turns the image back. Since the jaggies are horizontal-only for the reasons you described, I think you only need to:

EEDI2


and then scale vertically to 66.6...% and addborders/crop (depending on whether you crop initially or not) as necessary to get back to 480 pixels.

DE
Author
Time
Arnie, that is great work. Wish I knew how to do this stuff so I could do my own.

looking for HDTV of the  Attack of the Clones and Revenge of the Sith.  Also HDTV of The Lord of the Rings trilogy

Author
Time
@ Darth Editous
I experimented a bit and I think you are right, eedi2() will do. Now a just have to adjust the noise filter and sharpen filters.

@ fffffff
Thanks. But it is really not that difficult. I'm only applying what other people have come up with.
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
Can you post the final script when you're done?

If I had some gum, I’d chew a hole into the sun…

Author
Time
At the moment I'm trying someting with pixiedust (noise filter), eedi2 (to remove the jaggies) and 2 sharpening filters (limitedsharpenfaster for overall sharpening and fft3d for edge enhacement). Well, it's not a final version, schorman13 but here's what I have so far:


Loadplugin("dgdecode.dll")
MPEG2Source("Star Wars.d2v")

converttoyuy2
Loadplugin("loadpluginex.dll")
loadplugin("dustv5.dll")
PixieDust(limit=1)

loadplugin("eedi2.dll")
Crop(6,102,-2,-104,align=true)
eedi2()

loadplugin("fft3dfilter.dll")
FFT3DFilter(bt=-1, sharpen=0.6)

Loadplugin("mt_masktools.dll")
Loadplugin("warpsharp.dll")
Loadplugin("removegrain.dll")
LimitedSharpenFaster(dest_x=720,dest_y=360)

Addborders(0,60,0,60)
Converttorgb


I included the loading of the plugins in the script so you know what dlls are required.
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
EEDI2() works nicely on diagonal lines, but I haven't yet found a way to reduce the aliasing on edges which are near horizontal.

Anyway, what this thread needs is some more screenshots and some example scripts (so I can steal the script that gives the best result ). Here's my contribution (note, resized to 1:1 pixel aspect ratio for comparison):

http://img181.imageshack.us/img181/3153/originalvo9.png
Original

http://img168.imageshack.us/img168/6232/processedqx3.png
Processed

Script so far:
source=mpeg2source("e:\project.d2v").crop(6,102,-2,-104).eedi2()
backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
source.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)
limitedsharpenfaster(dest_x=720, dest_y=366)
addborders(0,57,0,57)
Arnie can you post the same frame (4451) from your script?

Guidelines for post content and general behaviour: read announcement here

Max. allowable image sizes in signatures: reminder here

Author
Time
even the processed one still has the motion blur on the rebel trooper running.

was that intentional in the filming?

or is it another part of the ugly dvnr?

the major problems as know on the pre special edition star wars films is the dirt and grain evidant in the scenes especially shot in tunisia that have sand and grit literally on the film.

“Always loved Vader’s wordless self sacrifice. Another shitty, clueless, revision like Greedo and young Anakin’s ghost. What a fucking shame.” -Simon Pegg.

Author
Time
I used virtualdubMpeg to make the screenshots, the images are to bright and I don't know how to change it (when I press play the brightness is normal).

This is without grain/noise removal:
http://img521.imageshack.us/img521/3636/x0000mn0.jpg

With pixiedust at "limit=1":
http://img253.imageshack.us/img253/7045/x20000vz9.jpg
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.
Author
Time
Arnie, there's very little difference between our two approaches (and your images are not too bright), although it looks like MVDegrain2 under standard usage (I just copied the example from the MV tools documentation) gives stronger degraining than PixieDust(limit=1).

Guidelines for post content and general behaviour: read announcement here

Max. allowable image sizes in signatures: reminder here

Author
Time
Edited because I'm tired and talking rubbish...

EEDI2 only searches a certain distance, which is effectively means up to a certain angle. You can increase the parameter, but it will still only go so far.

DE
Author
Time
Could someone post the filters MVDegrain2 and MVAnalyse that are being used to do this? I have looked, but can not find them. If someone could post them it would be great!


Author
Time
EEDI2().TurnRight().EEDI2().TurnLeft() is indeed one of the best antialiasing filters on the Avisynth side. For extreme cases, preceding it with AAA() (it's an avisynth function, you need AAA.avs, UnFilter.dll and SangNom.dll) produces awesome results, but the problem with AAA() is that it blurs the picture too much.

Moth3r, good call using MVDeGrain2, it's an awesome denoiser. By the way, you can increase its effectiveness if you feed its MVAnalyse portion a denoised input in this manner:

source=last
denoised=YOUR_DENOISER
backward_vec2 = MVAnalyse(denoised,isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = MVAnalyse(denoised,isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = MVAnalyse(denoised,isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = MVAnalyse(denoised,isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
MVDegrain2(source,backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=2)

(my preferred denoiser is DeGrainMedian FWIW)

I never heard of horizontal aliasing though, are you sure it's not dotcrawl?
Author
Time
Thanks Byakko; I'll try using MVDeGrain2 in conjunction with DeGrainMedian and see what the results look like.

The aliasing I was referring to exists on diagonal edges at a shallow angle, (i.e. almost horizontal). EEDI2() is not effective on such aliasing. I'll find an example later and post a screenshot.

Guidelines for post content and general behaviour: read announcement here

Max. allowable image sizes in signatures: reminder here

Author
Time
Like these lines?
http://img211.imageshack.us/img211/9601/0000oa6.jpg
Fez: I am so excited about Star Whores.
Hyde: Fezzy, man, it's Star Wars.