Info: Digging up those blacks - using the STAR WARS Blu-ray for preservations

Hey guys, I’m a sort of new guy here in a sense, though I’ve posted here and there for some time and been reading the forum since years ago.

The recent years I’ve become interested in home cinema calibration and related things, which brought me into color correction. I’m not at all anything close to professional in these things though, but I have many years of photoshop experience.

When I rediscovered this forum a couple months ago, I was instantly amazed at the work many people here are doing to preserve the original trilogy. I saw that since there’s no high-res source of the movies with intact shadow details, people have been forced to base a lot of stuff on the HDTV versions. I’ve been fiddling around myself in photoshop with some raw screenshots of the HDTV versions but since they’re compressed by broadcast, the shadow detail has suffered.

Now that the Blu-ray set was released, I thought I’d do that again and this time, since there was not much compression, there was also slightly more shadow detail. I went into photoshop and made a curve file and some settings using “selective color”, to try and dig up some shadow detail and make like “one setting for the whole movie” to remove the largest color inaccuracies and tints etc. I also tried to correct the underexposed look of the SE’s. (Only tried with SW so far)

Now, I know that at least Harmy has gone through this kind of thing, using the HDTV version as a base. (Totally love your DE versions Harmy) I found out that Adobe after effects could use both curves and selective color, so I took the raw Blu-ray file of SW and did a render using my settings.

The reason for this thread is not that I want to make an own preservation, but maybe “help” others by posting my settings.

Note that my monitor is not displaying 100% correct colors, and that I, like everyone else, get a bit colorblind after looking at these things for a while. That said, skin tones might not be all correct, but at least it’s not as much “lobster-man” now. After I post the settings, maybe others with better eyes can improve them even more.

Finally, sorry in case my grammatics is not perfect, I’m swedish after all…

EDIT: New screenshots! These shots represent the latest avisynth settings which you can find in post 3. They are the exact same shots as the ones in the screenshot thread. They were rendered using the built in imagewriter plugin in avisynth.

You_Too: (Don't know how to make nice thumbnails here)

When you upload an image to imageshack.us they make the code for a thumbnail version.  Under the image -> embed thumbnails of this image -> HTML, copy that code and 'Preview' in the OT.com post:  will look like this:

  (there's a different link option which will show the image size in the thumbnail.)

Actually, the HTML option didn't work for some reason. But today I managed to use another method, so now the thumbnails are up.

Also, here's my settings: http://www.multiupload.com/CBAS7KQS9O

It's the curve file and selective color file. The curve file can be opened in both photoshop and after effects, but the selective color file can only be used in photoshop, at least I think so.

So here comes the selective color settings. (Important: Use selective color BEFORE curves. At least that's what I did.)

Reds: Cyan +50%, Yellow +7%
Yellows: Cyan -25%, Yellow +15%
Greens: Magenta +100%
Cyans: Cyan -25%
Blues: Yellow +75%
Magentas: Cyan -25%, Magenta -75%, Black +100%

Some will notice that the curve file will push blacks down from 1 to 0. That's because in the end I found it looked better this way. It was like the blacks on the Blu-ray was all at 1 instead of 0.

(THIS POST WILL BE EDITED WHEN SETTINGS ARE IMPROVED)

Avisynth version!

For you who like me prefer avisynth to render your movies, I've got good news. I managed to change the magenta setting so it didn't need "black +100%", so we're now able to use the selective color plugin for avisynth.

HUGE thanks to Darth Editous for the plugin. Or well, the author who by a strange coincidence have never been seen in the same room as Darth Editous at the same time...

For convenience, I've packed the plugins you need (Gicocu, selective color) as well as the curve (modified to GIMP format for Gicocu) and a script example.

Download link: http://www.multiupload.com/93LYOPT7C8

For those new to this, a good example of a program that can render movies using avisynth is Megui. Avisynth is installed with Megui and usually ends up in C:\Program Files (x86)\AviSynth 2.5\ and in there you can find a folder named plugins, that's where Gicocu and selective color should be. The movie and curve file should be in the same folder when using the script.

Script example:

DirectShowSource("Starwars.mkv")

ConvertToRGB32(matrix="rec709")

selectivecolour(last,\
0.5,0,0.07,\
-0.25,0,0.15,\
0,1,0,\
-0.25,0,0,\
0,0,0.75,\
1,-0.5,1,\
false)

GiCoCu("sw13.cur")

ConvertToYV12()

Change Starwars.mkv to whatever your movie file is named. Myself I remuxed the crawl and rest of the movie together in an mkv file and that's why it's named like that.

If you want to remove the black borders from the movie when rendering, add the line:

crop( 0, 131, 0, -131)

If you want to render the movie without audio to remux it afterwards, change the first line to this for example:

DirectShowSource("Starwars.mkv", fps=23.976, audio=false, convertfps=true).AssumeFPS(24000,1001)

And to those really into color correcting: The last line should be "ConvertToYV12()" and not have "(matrix="rec709")" added. I tried this first but the colors were messed up. It was enough to have it at the first color conversion line.

NEW EXPERIMENTAL ADD-ON SCRIPT:

Add this at the bottom of the script. Reduces dark blue colors in the shadow details. (Requires Masktools plugin for avisynth)

Function SelSah_v3 (clip clp, float "Sat", float "Hue", int "Luma_Low", int "Luma_High", int "Low_OS", int "High_OS",
\                    Float "minSat", float "maxSat", float "startHue", float "endHue", float "interp", clip "clp2", 
\                    bool "Exclude", bool "Preview") 
{

#  Extends Tweak to allow selective control of saturation and hue over a defined luma range. 
#  Assembled by WorBry. MaskTools computations kindly provided by Didee. Thanks to Gavino for advice on script syntax.   
#  Requires YV12 clp input and assumes full (pc) luma range 0-255.
#  Requires MaskTools v2. Tweak should be v2.58 or higher.
#
#  Parameters:
#  Clp       - Is the input clip that will be processed.
#  Clp2      - Provides option to use a second clip with different luma curve for the saturation/hue control.
#  Sat       - Saturation. Desaturation 0.0 =< 1.0 > Increased saturation. See Tweak 2.58 doc 
#  Hue       - Hue. Range -180 to +180. Default 0.0. See Tweak 2.58 doc 
#  startHue  - See Tweak 2.58 doc for details. Range 0 - <360. Default 0 
#  endHue    - See Tweak 2.58 doc for details. Range >0 - 360. Default 360
#  minSat    - See Tweak 2.58 doc for details. Range 0 - <150 ; must be < MaxSat. Default 0
#  maxSat    - See Tweak 2.58 doc for details. Range >0 -150. Default 150
#  interp    - See Tweak 2.58 doc for details. Range 0 - 32. Default 16
#  Luma_Low  - Lower luma range point. Range 0 - 255*  Default 0
#  Luma_High - Upper luma range point. Range 0 - 255*  Default 255
#  Low_OS    - Lower over-shoot point. Provides soft roll-off at lower range boundary. Range 0 - 255*
#              Default is Luma_Low+20 for normal (inclusive) range mode and Luma_low-20 for Exclude mode. 
#  High_OS   - Upper over-shoot point. Provides soft roll-off at upper range boundary. Range 0 - 255*
#              Default is Luma_High-20 for normal (inclusive) mode and Luma_High+20 for Exclude mode.
#  Exclude     True/False. True excludes the range selected by the luma set points. False (default) includes the selected range.  
#  Preview   - True/False. True gives preview of input clip, output and luma masks with original and output chroma.
#              Default is false
# 
#  *Note: the luma set points are limited with respect to each other to ensure correct orientation and avoid overlap.   
#
#Defaults:
Sat       = default(Sat, 1.0)
Hue       = default(Hue, 0.0)
startHue  = default(startHue, 0)
endHue    = default(endHue, 360)
minSat    = default(minSat, 0)
maxSat    = default(maxSat, 150)
interp    = default(interp, 16) 
Exclude   = default(Exclude, False)
Luma_Low  = default(Luma_Low, 0.0)
Luma_High = default(Luma_High, 255) 
Low_OS    = Exclude ? default(Low_OS,  Luma_Low + 20)
\         : default(Low_OS,  Luma_Low - 20)   
High_OS   = Exclude ? default(High_OS, Luma_High - 20)
\         : default(High_OS, Luma_High + 20)
clp2      = default(clp2, clp)
Preview   = default(Preview, False)        
#
#Limits: 
Sat       = Sat <0.0 ? 0.0 : Sat
Hue       = Hue <-180 ? -180 : Hue > 180 ? 180 : Hue
Luma_Low  = Luma_Low  <0.0 ? 0.0 : Luma_Low  >255 ? 255 : Luma_Low
Luma_High = Luma_High <0.0 ? 0.0 : Luma_High >255 ? 255 : Luma_High
Low_OS    = Exclude &&  Low_OS \         : Exclude &&  Low_OS > High_OS ? ((Luma_Low + Luma_High)/2)                 
\         : Low_OS  <0.0 ? 0.0 : Low_OS >Luma_Low ? Luma_Low : Low_OS      
High_OS   = Exclude &&  High_OS >Luma_High ? Luma_High
\         : Exclude && High_OS < Low_OS ? ((Luma_Low + Luma_High + 2)/2)
\         : High_OS 255 ? 255 : High_OS
#
#Limits for startHue, endHue, minSat and maxSat set by Tweak 
#
#Create Saturation and Hue control clip:
twclp = clp2.tweak(sat=Sat, hue=Hue, startHue=startHue, endHue=endHue,
\       minSat=minSat, maxSat=maxSat, interp=interp, coring=false)
#
#Create luma mask with low/high range points and roll-off gradient at boundaries:  
#"Include" [0]<-black->[Low_OS]<-gradient->[Luma_Low]<-white->[Luma_High]<-gradient->[High_OS]<-black->[255]
#"Exclude" [0]<-white->[Luma_Low]<-gradient->[Low_OS]<-black->[High_OS]<-gradient->[Luma_High]<-white->[255]
#
Low_OS    = string(Low_OS) Luma_Low  = String(Luma_Low)
Luma_High = String(Luma_High) High_OS = String(High_OS)
#Create "Include" range expression for mask:
blkI      = "x "+Low_OS+" < x "+High_OS+" > | 0 "
whtI      = "x "+Luma_Low+" >= x "+Luma_High+" <= & 255 "
grLI      = "x "+Luma_Low+" < 255 x "+Low_OS+" - "+Luma_Low+" "+Low_OS+" - / * "
grHI      = "x "+Luma_High+" > 255 1 x "+Luma_High+" - "+High_OS+" "+Luma_High+" - / - * "
exprI     = blkI + whtI + grLI + grHI + "x ? ? ? ?"
#Create "Exclude" range expression for mask:
blkE      = "x "+Low_OS+" >= x "+High_OS+" <= & 0 "
whtE      = "x "+Luma_Low+" < x "+Luma_High+" > | 255 "
grLE      = "x "+Low_OS+" <= 255 x "+Low_OS+" - "+Luma_Low+" "+Low_OS+" - / * "
grHE      = "x "+Luma_High+" < 255 x "+High_OS+" - "+Luma_High+" "+High_OS+" - / * "   
exprE     = blkE + whtE + grLE + grHE +  "x ? ? ? ?"
#Luma Mask (greyscale for output):
lmask     = twclp.mt_lut(expr = (Exclude ? ExprE : ExprI))
lmaskg    = lmask.Greyscale()
# Apply luma mask:
Output    = mt_merge(clp,twclp,lmaskg,luma=true,Y=2,U=3,V=3)
#
#PREVIEW set-up
clps      =clp.Subtitle("Original Clip").LanczosResize(640,360)
outs      =output.Subtitle("Final Output").LanczosResize(640,360)
#Luma Mask with chroma from tweaked clp
lmcts     = lmask.Subtitle("Luma mask + Tweaked Chroma").LanczosResize(640,360)
#Luma Mask with chroma from original clp
lmcc      = Exclude ? clp2.mt_lut(expr=ExprE) : clp2.mt_lut(expr=exprI) 
lmccs     = lmcc.Subtitle("Luma Mask + Original Chroma").LanczosResize(640,360)
#Create composite view
Clpout    = StackVertical(clps,outs)
Masks     = StackVertical(lmccs, lmcts)
Prev      = StackHorizontal(Clpout,Masks)
#
Final     = Preview ? Prev : Output
#
Return(Final)

}

SelSah_v3(Luma_Low=0, Luma_High=25, startHue=310, endHue=80, Sat=0.75)

 

I like your colors. They're still not 100% correct, but better than on the official releases

Thanks! Yes, 100% correct would be very hard to achieve, and would of course require a shot-by-shot correction. I was just aiming for an overall correction.

Though I'm not sure about cyan yet, until I've checked how it affects the death star's walls.

Looks dang good, it does.

Hah, I didn't notice before that the escape pod hatch is still blue when the Stormtroopers are looking it over....

Added some more shots from the beginning of the movie. For some reason this part was separated from the rest on the Blu-ray, I guess it's because some countries translated the crawl and had a custom one made which was put there.

R2 and 3PO on Tatooine - gah, more "watercolor filter" DNR there. R2 looks he was detailed by Claude Monet. (Again, as You_Too said in Harmy's thread, not sure if the grain removal was always this egregious but it's just more apparent now.)

And look at how they took the blue out of the hatch, it looks like all of the dark areas in the impact crater and tracks are the same odd shade of dark grayish mauve. In fact, I can see that same shade in a lot of the shadow details. (It's been discussed inside out for the past 7+ years, but I ask yet again, what's with all the weird colors in the highlights and shadows in the '04/'11 transfer? Dark areas that looked black or neutral before suddenly look blue, purple, etc...)

It still makes my head spin - why is it that some shots look amazing, while others look like absolute rubbish? Why do some shots seem to have intact grain structure, full dynamic range and accurate color values, while others have severe degraining artifacts, crushed blacks, clipped whites, and bizarre hues that seem like they don't exist on this planet? How is it that the exact same transfer can go from scenes like "Look sir, droids" and "I suppose you're programmed for etiquette and protocol" that make me feel almost like I'm looking at the original negative, to wax-sculpture droids, purple shadows, lobster-men and a Sandcrawler interior with featureless black voids that looks like it's getting swallowed up by the Nothing?

You_Too, I definitely like your color correction. The lower contrast and warmer timing look very nice and evoke a retro feeling. In short, I like it in general. When Star Wars showed in theaters, it was probably a tad bit more saturated and cooler, but, as far as being aesthetically pleasing, yours look nice. Good job!

Thanks for all nice comments on the colors. About them being "warm", I also think it suits the movie well, and some scenes still look "cool". Compare the two shots of Leia in front of the stormtroopers. In the close-up one, the whites look slightly more blue.

Yeah maybe. Though there are some shadow detail that was pushed down too much on the Blu-rays that I managed to bring back up a bit using my curve setting, there's still some missing detail which can't be recovered like you say.

That said, is there any way to rename the thread? :)

Yeah, but a lot of that would require shot-by-shot correction. I'm trying to create the "best" overall settings to improve the picture as much as possible without a shot-by-shot correction.

By the way, right now I'm trying some new settings for magenta in selective color. Take a look at the screenshot of Luke looking at Beru, and look closely on the dark parts of Luke's hair. Of course that should be brown and not have a tint of magenta/purple. If going by how the GOUT looks, it should also be slightly brighter. I think it's possible to correct that, and if I succeed I will post new screenshots and settings.

Allright! I tried a new setting for magenta in selective color (cyan -25%, magenta -75%, black +100%) and here's the result:

Here's a comparison of Luke's hair to my old settings. (New on right)

The new settings of course affected other scenes as well. Here's something very visible it did to the pink flashes here and there. New settings on top, old on bottom.

It also made the skin tones of the "lobster-man" look better. (New on right)

Please tell me what you think!

You_Too said:

Please tell me what you think!

It's a noticeable improvement, especially with the flashes. Keep up the good work, You_Too. I like your color corrections.

These images look quite good so far.  It seems like you've really locked in on exactly what sort of manipulation was done to the colours and contrast, and undone it as precisely as could be hoped for with general settings.  Of course, many shots would still need individual correction, due to things like desaturated laser blasts and intentional colour changes—for example, the red on the blockade runner was a dark shade of maroon in the original version, while even with the correction it still shows up as bright red due to the 2004 change.

Looking forward to seeing what else you can accomplish with this.  Is there a way to replicate these settings with AviSynth?  I'd be very interested in that.

hairy_hen said:

These images look quite good so far.  It seems like you've really locked in on exactly what sort of manipulation was done to the colours and contrast, and undone it as precisely as could be hoped for with general settings.

That's what I'm trying to do, at least to undo the worst changes.

hairy_hen said:

Is there a way to replicate these settings with AviSynth? I'd be very interested in that.

Unfortunately, you can only use curves in Avisynth, and it's a bit more complicated since it requires a plugin called gicocu, and then you have to input the curves into another program called GIMP and save them from there, for gicocu to be able to use them. On top of that, GIMP doesn't bend the curves like photoshop, so you'd need to add more anchor points in the curve for it to end up the same.

The worst thing is that there is no selective color plugin for Avisynth, at least that I could find with a lot of googling. There is a plugin called Tweak, but you can only change hues of colors in that one, and the result doesn't come anywhere close to selective color.

So the only solution so far is to use Adobe after effects for rendering this using my settings.

Asaki said:

You_Too said:

If going by how the GOUT looks...

Sounds like a bad idea =|

I said that because GOUT has a bit more "natural-looking" colors, and some more shadow detail. It's undersaturated though so of course I can't use it as pure reference.

Anyway, here's another comparison of the color errors the magenta setting fixed. (Old on top)

Right now I'm working on eventually changing the setting for cyan, because for example it seems it can fix Leia's hologram. I would like to request something to use as a reference, like 35mm scans or something.

You_Too write: That said, is there any way to rename the thread? :)

When you edit the first post, the thread title can be renamed.

Yay, you got rid of the purple shadows! I've always been baffled at all the purple shadow detail on Tatooine in the '04 transfer. It's like the pink-shifted 1997 transfer, except that not all of the colors are affected...

The worst thing is that there is no selective color plugin for Avisynth, at least that I could find with a lot of googling.