I've found that DeHalo_alpha() is non-intuitive. It's best to try the default settings and then test-change individual numbers to see the effect. It also helps to add a magnified view in Avisynth for a better look at what's happening.
The 2 top pictures are the before & after compare using DeHalo defaults. It's not bad but halo-ing still shows. Just use DeHalo_alpha() without any settings for that.
The bottom picture is the after part of my trying a few tweaks for stronger fixing. That one is better and halo-ing doesn't show much at all.
- The "default radius" operations for x (horizontal) and y (vertical) are "2". As halo-ing doesn't much show vertically, set ry=1.0 for minimum (or none?) vertical processing.
- The "default strength" operations on dark and bright halos are "1.0". Set darkstr=1.5 and brightstr=1.5 for my stronger and better result. These seem to work best at the same value. Be sure to make small adjustments or ugly artifacts will appear.
DeHalo_alpha() does tend to flatten the contrast so I also added Levels() to darken the now-not-so-dark areas and brighten the now-not-so-bright areas. Always verify to not crush nor blow-out your luminance.
I had previously worked up this AviSynth script to do just this. As you're still a novice at this, here's my (simplified) script with which to experiment. Just comment/un-comment/change the lines you need to, for your tests.
## THX1138 DeHalo_alpha demonstrator
##=================
## Video path & name
clip = "THX1138 US WS 1991 LD.png"
## Get video / picture
#v1 = DirectShowSource( clip ) # for video
v1 = ImageReader( clip ) # for picture
# video / picture size
x_size = 640
y_size = 286
# original & fixed streams for processing
v1_fix = v1
##----------
## DeHalo_alpha()
## clip clp, float "rx", float "ry", float "darkstr", float "brightstr",
## float "lowsens", float "highsens", float "ss"
#
### rx, ry [float, 1.0 ... 2.0 ... ~3.0] Default 2.0
# As usual, the radii for halo removal.
# this function is rather sensitive to the radius settings. Set it as low
# as possible! If radius is set too high, it will start missing small spots.
### darkstr, brightstr [float, 0.0 ... 1.0] Default 1.0, 1.0 allowed
# The strength factors for processing dark and bright halos. Default 1.0 both
# for symmetrical processing. On Comic/Anime, darkstr=0.4~0.8 sometimes might
# be better ... sometimes. In General, the function seems to preserve dark
# lines rather good.
### lowsens, highsens [int, 0 ... 50 ... 100] Default 50
# Sensitivity settings, not that easy to describe them exactly ... in a sense,
# they define a window between how weak an achieved effect has to be to get
# fully accepted, and how strong an achieved effect has to be to get fully
# discarded.
### ss [float, 1.0 ... 1.5 ...] Default 1.5
# Supersampling factor, to avoid creation of aliasing.
##----------
# DeHalo (calls the function script at bottom)
v1_fix = v1_fix.ConvertToYV12() # DeHalo uses this format
#v1_fix = v1_fix.DeHalo_alpha( rx=2.0, ry=2.0, darkstr=1.0, brightstr=1.0 ) # defaults
v1_fix = v1_fix.DeHalo_alpha( ry=1.0, darkstr=1.5, brightstr=1.5 ) # tweaks
v1_fix = v1_fix.ConvertToRGB24() # return to default format
v1_fix = v1_fix.Levels( 24, 1.1, 248, 0, 255, coring=false ) # fix lost contrast
## display results
# add 4x magnify panel to original
v1_mag = v1
\.PointResize( x_size*4, y_size*4 )
\.Crop( x_size*10/5, y_size*10/6, x_size, y_size )
\.Crop( x_size/3, 0, x_size/2, y_size )
v1 = StackHorizontal( v1, v1_mag )
# add 4x magnify panel to fix
v1_fix_mag = v1_fix
\.PointResize( x_size*4, y_size*4 )
\.Crop( x_size*10/5, y_size*10/6, x_size, y_size )
\.Crop( x_size/3, 0, x_size/2, y_size )
v1_fix = StackHorizontal( v1_fix, v1_fix_mag )
# display before & after split screen
StackVertical( v1, v1_fix )
##==========
## Functions
##----------
##==========
## De-halo
# Here are the parameters:
# - rx, ry [float, 1.0 ... 2.0 ... ~3.0] Default 2.0
# As usual, the radii for halo removal.
# Note: this function is rather sensitive to the radius settings. Set it as low
# as possible! If radius is set too high, it will start missing small spots.
# - darkkstr, brightstr [float, 0.0 ... 1.0] Default 1.0, 1.0 allowed
# The strength factors for processing dark and bright halos. Default 1.0 both
# for symmetrical processing. On Comic/Anime, darkstr=0.4~0.8 sometimes might
# be better ... sometimes. In General, the function seems to preserve dark
# lines rather good.
# - lowsens, highsens [int, 0 ... 50 ... 100] Default 50
# Sensitivity settings, not that easy to describe them exactly ... in a sense,
# they define a window between how weak an achieved effect has to be to get
# fully accepted, and how strong an achieved effect has to be to get fully
# discarded.
# - ss [float, 1.0 ... 1.5 ...] Default 1.5
# Supersampling factor, to avoid creation of aliasing.
function DeHalo_alpha(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss")
{
rx = default( rx, 2.0 )
ry = default( ry, 2.0 )
darkstr = default( darkstr, 1.0 )
brightstr = default( brightstr, 1.0 )
lowsens = default( lowsens, 50 )
highsens = default( highsens, 50 )
ss = default( ss, 1.5 )
LOS = string(lowsens)
HIS = string(highsens/100.0)
DRK = string(darkstr)
BRT = string(brightstr)
ox = clp.width()
oy = clp.height()
uv = 1
uv2 = (uv==3) ? 3 : 2
halos = clp.bicubicresize(m4(ox/rx),m4(oy/ry)).bicubicresize(ox,oy,1,0)
are = yv12lutxy(clp.expand(U=uv,V=uv),clp.inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
ugly = yv12lutxy(halos.expand(U=uv,V=uv),halos.inpand(U=uv,V=uv),"x y -","x y -","x y -",U=uv,V=uv)
so = yv12lutxy( ugly, are, "y x - y 0.001 + / 255 * "+LOS+" - y 256 + 512 / "+HIS+" + *" )
lets = maskedmerge(halos,clp,so,U=uv,V=uv)
remove = (ss==1.0) ? clp.repair(lets,1,0)
\ : clp.lanczosresize(m4(ox*ss),m4(oy*ss))
\ .logic(lets.expand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"min",U=uv2,V=uv2)
\ .logic(lets.inpand(U=uv,V=uv).bicubicresize(m4(ox*ss),m4(oy*ss)),"max",U=uv2,V=uv2)
\ .lanczosresize(ox,oy)
them = yv12lutxy(clp,remove,"x y < x x y - "+DRK+" * - x x y - "+BRT+" * - ?",U=2,V=2)
return( them )
}
function m4(float x) {return(x<16?16:int(round(x/4.0)*4))}
##==========