Spaced Ranger

Thanks! And glad to help.

I'm noticing that some posted pictures are having trouble getting through from ImageShack, from time to time. (They must be swamped with requests and just drop those they can't handle.) Be sure to set your browser to display "image placeholders" so you'll know if you're not getting any required pictures.

Yes, definitely!

** BTW, a battery back-up gives much piece of mind. There's nothing like an orderly computer shutdown after the power goes out. ;) **

In anticipation of color correcting, I've been doing more concept brainstorming. Then I noticed that the paint program's curves results (from sampled points) looked familiar -- like when using the histogram expand and compress adjustments. This intrigued me as I'm still researching the "fade-fingerprint" (how film fades over time) technique for fade auto-compensation.

By using "midtones: expand / compress" (so named in this paint program) to individually restore the fade-flattened ranges of the RGB separations, and "gamma" to re-center the ranges when needed, I was able to manually step-back the fade ... in the way it faded (very pleased about that).

In the histogram function, the grey is the original fade damage and the resultant red-overlay is the correction. It is important to note that I made anti-fade adjustments by concept and only used the eye-dropper to verify high-end and low-end fine tuning:

The result is the best yet -- which included a pre-smoothing (an improvement over the post-smooth of my previous post), which prevented "new exaggerated grain" from forming in the first place:

Hopefully, this will translate well when using video tools.

captainsolo said:

THX never ran on TV. ...

Just FYI ... "Columbia Broadcasting System (CBS) (1973) (USA) (TV) (original airing)" -IMDb: THX 1138

BTW, anyone have a taping of such early broadcasts (consumer recording formats: Betamax 1975+, VHS 1976+)?

Your's is a pretty good color-correction (much better than the faded source).

I, too, have played with assorted settings and just cannot get it spot-on. In past, I would resort to a cascade of filtering [in paint program tests] to get it better. (Not happy using that approach, though. Being a minimalist, I always search the core of the process for the problem and a "magic bullet" to kill it with one shot.)

Your recently posted, corrected video segment has the brightness varying within individual shots. How did you capture the original 16mm film (forgive the ignorance -- my quick scan of this sizable thread is confusing as you seem to be working on a number of preservations)? Is it a capture problem? (BTW, do you test & adjust for a non-blown-out/crushed capture?) Will you leave it in that condition? How about the all the per-frame dirt/spots and full-length scratches? (Only asking because color correction should be performed on a stable source.)

Reviewing this previous "color corrected" frame reminded me to bring up a caveat. Notice the "grain" in SEN's face? That is not film grain. Rather, it is by-product of color correction in the digital world. For example, consider the Star Trek book Star Trek The Original Series 365:

It started out as a Star Trek fan effort to collect and archive "lost" 35mm Star Trek film clips, with an eye to possible publication (their updated website StarTrekHistory.com removed most of the original altruistic sharing of their archive when they went commercial). Interestingly, many of the faded 35mm film clips, now "restored", had an odd "signature" grain ... the same grain that appears in SEN's restored frame here. Same "grain", same "problem".

Film fade narrows the brightness range of the 3 color layers, to different degrees, with this recognized result:

In RGB, red takes the most hit. When RGB separated, this effect from the original faded frame can be seen in a very flat-looking red channel:

The aim of color correction is to spread that range back out to cover it's original spectrum:

But "new exaggerated grain" is generated from the "quick & dirty" digital process and is even visible in the representative graphs -- the smooth original curve to the stepped corrected one. Those steps must be smoothed.

Normally, smoothing would be applied to the complete frame, but that reduces legitimate detail, too, throughout the entire picture. However, while in RGB separations, the red (most correction) and the green and the blue (least corrections) can be smoothed individually, judiciously, and only as strong as needed in each case -- keeping as much real detail as possible. So, I applied a fairly strong "edge-preserving smooth" to red, a very weak one to green, a slightly stronger one to blue, and recombined the RGB separations to:

which now compares well to the non-corrected videotaped broadcast:

Keep in mind this demonstration was made in a paint program and does not exactly mimic video processing ... for better or for worse. :)

msycamore said:

The approach was just me playing around with colormill in VirtualDub ...

It's very hard to twiddle dials and eye-ball it. Every adjustment affects everything else, and the mind's eye further interferes with the perception of the results. We need something more definitive ... like a fade-fingerprint™ of a film's present state of deterioration to calculate a counter-fade self-correction. But until then ... cheat! :)

msycamore said:

Also, if you Spaced Ranger or anyone else is willing to give the 16mm footage a shot with your methods, let me know ...

Sure! That'd be cool! (PM me with details.)

SilverWook said:

Never knew there were any widescreeen showings on cable. I can't recall even seeing it on TCM before the DE came along. Same damn video master...

This particular TNT fullscreen was cablecast 1995 and the Bravo widescreen one was cablecast 1997.

.

@msycamore - BTW, have you posted a raw video clip of this last-posted SEN scene?

msycamore said:

I played around with the 16mm footage today to see if I could get it to look somewhat acceptable, here's my result on "The Lost Scene." http://www.sendspace.com/file/twyqdo

Let me know what you think. Ignore the audio, at the moment I don't know what the heck is causing it to get out of sync!

Before

After

I'm afraid there's not that much color left to work with but I do think it's watchable in this state, maybe Puggo or someone else with more experience than me when it comes to color correcting could get it to look even better for an eventual "grindhouse DVD."

It's an improvement but a more effective approach is needed (BTW, what is your approach?) to make the best of the remaining faded color (in RGB, the red-blue suffers the most from fade). For example, this is a pretty tough frame on which to adjust white balance because there are no visible highs, mediums, and lows (yes, all three) of guaranteed grey objects on this source, except for bright white uniforms.

However, we can cheat (oops, did I say that?) and use as guide an original print. In this case, I capped old broadcasts from TNT (fullscreen) and Bravo (widescreen) -- say "hello" Coilly (nyuk, nyuk, nyuk):

Well, what do you know, that background wall was originally grey! Cheats are good! My personal preference for accurate color is of the widescreen, but for sake of A-B comparison, I've used the fullscreen for this demonstration.

First, locate high (1), medium (2), and low (3) originally-grey-areas to sample:

Of course, getting a good grey-scale spread is key to good color-correction. Sometimes, one simply cannot use a single frame and must sample across multiple frames (of the same scene) to get the needed spread. Also, a sample will be better if taken by an "eye-dropper" that can grab a radius of pixels for an average value (minimizes noise errors). The following are RGB values pulled from the 16mm source and TNT target areas:

R ... G ... B
----- ---- ----
157 218 183 - 1 "high" source area
201 198 212 - 1 "high" target area
----- ---- ----
129 146 129 - 2 "medium" source area
123 122 127 - 2 "medium" target area
----- ---- ----
112 078 083 - 3 "low" source area
085 085 085 - 3 "low" target area

Then use the source-target points for input-output values to skew the R-G-B scales for color compensation:

And, yes, it works:

If the color-fade is mostly uniform throughout the 16mm print, a single/best adjustment (or sampled RGB points, from across the entire movie, averaged together) may suffice for the entire film.

SilverWook said:

What's equally amazing is how little publicity the changes got compared to the Special Editions. ... This also seems to be the beginning of the trend to not even credit the people who worked on the new shots.

msycamore said:

... What surprises me the most, is the extreme re-editing that took place, I simply cannot understand the reason for it other than it was done just for the sake of re-doing it.

Unfortunately, the reason is ... who George Lucas is. Once you know -- "I'm a '60s, West Coast, liberal, radical, artsy, dyed-in-the-wool 99 percenter before there was such a thing" [George Lucas, 1/18/2012] -- his being a 1984-ish communist-party-faithful explains it all (see http://originaltrilogy.com/forum/topic.cfm/My-STAR-WARS-Thesis-I-need-help/post/569749/#TopicPost569749 ).

"Nikolai Yezhov, walking with Stalin in the top photo from the 1930s. Following his execution, Yezhov was edited out of the photo by Soviet censors. Yezhov became an 'unperson'." - Wikipedia: Nineteen Eighty-Four . No mention of there being an "edit"; no credit to the "editor", either.

And so ... the original THX 1138 has become an "unmovie". :)

To paraphrase Group Capt. Lionel Mandrake (Dr. Strangelove): "It was just Lucas's way of having a bit of fun, the swine. Strange thing is he makes such bloody good movies."

Laserschwert said:

Weeell, I think you're underestimating the work I put into this. ...

I wasn't minimizing your work at all, which sounds great! In fact ... listening to it right now ("Main Hangar"). :) Rather, I was just adding to the conversation, with the MIDI to wavetable hardware available (the Gravis UltraSound was supposed to be the first & best) even when gaming music was in it's infancy.

Your website is awesome, too (going to start perusing it soon)! You really should add that link to your signature. And maybe post more details here on your process, software, hurdles, tips-&-tricks, etc.?

Laserschwert said:

This was kind of an on and off project for me, but in the last few days I finally started over from scratch and finished this thing:

TIE Fighter - Original Videogame Soundtrack

Excellent work! You know, LucasArts has lots of MIDI-music games to keep you busy ... :)

Yet, even back in the day of such MIDI-music games as TIE Fighter, many must've played those games sounding like your project ... with this:

Wikipedia - Gravis Ultrasound :

The Gravis UltraSound was unique at the time of its launch (1992) with its use of 'wavetable' sample-based music synthesis technology on the IBM PC platform — the ability to use real-world sound recordings rather than artificial computer-generated waveforms to base a musical instrument on; so a piano sounds like an actual real piano, a trumpet like an actual trumpet, etc. The GUS was remarkable for MIDI playback quality with a large set of instrument patches that could be stored in its own RAM, having up to 32 hardware audio channels.

Here's an old advert:
http://www.csoon.com/issue1/gravis_h.html

An ode to UltraSound (with "the original UltraSound & samples CD" link):
http://multimedia.cx/eggs/ode-to-the-gravis-ultrasound/

And a GUS [Gravis UltraSound] emulator(!):
http://www.deinmeister.de/gusemu/index.html

g-force said:

Here's some screencaps using the 04 HD broadcast and the above script:

http://imageshack.us/g/11/o4color2avssnapshot0000.png/

-G

Although I prefer the concept of working on RGB-curves, I like the look of your new results ... except that the colors are too saturated. A reduction of about 15% should remove the effect of "green-tinge", or "yellow-tinge", or other tinging (depending on the shot), while still letting your colors "pop" (as movie reviewers like to call it). Also, the overall improved brightness nicely opens up details hidden in the darker areas.

snicker said:

Look out for a dedicated thread in a couple of days.

Impressive! And looking forward to it, as I'm sure it will be an awesome technical resource on the OT forum.

Harmy (and crew) is doing a great job with spectacular results on his "despecialization" v.1 & v.2. But despecialization is not only restoring missing shots and removing new shots & shot-elements -- although that is a big and obvious part. Whether by sloppiness-commanded or sloppiness-approved, George Lucas has ruined the Star Wars picture, generally, in his post-production "specialization", with the blown out hi-lights, the crushed shadows, and the shifted colors ... even in isolated color-layers ... as a side-effect of his company's incompetent toying with "grading". (Side note: this seems to be an industry fetish now-a-days, as many different forum postings have attested to -- Star Wars, The Lord Of The Rings, et al.)

Although it may not be in mind for this project (didn't Harmy also do a "partial-despecialization" once?), "full-despecialization" would include these corrections. Hopefully it will be included ... maybe in v.3? ;)

snicker said:

I've been working with individual colour channels to build up detail in each one. The red channel highlights clip a lot, the blue channel too but not quite so much and the green channel rarely clips. It's not often that you have clipping in all three channels simultaneously so there is usually good detail in at least one channel. The green channel is often useless because the white point is so much lower than the other channels but it can be used as a 'base' to restore some detail in combination with the blue channel which contains most of the highlight detail.

I've used 4 keys just in the red channel alone to build up the red highlight detail as well as using difference mattes to limit the areas which are being altered.

So basically I started with three precomps - one each for the red, green and blue channels, worked on building the detail in each using colour and luma keys, then used 'set channels' in a fourth precomp to set each of them as new R/G/B channels. This then becomes your new video layer.

Scanned through the thread and came across this. I've been thinking of using multiple separations for various reconstructions (the data is there, we just have to know where to look ... to paraphrase The X Files) and your idea here is brilliant. Two thumbs up!

Spaced Ranger said:

... (I vaguely remember seeing one, some time ago, somewhere).

I know that wasn't very helpful. So I found the references (have never implemented them myself ... but will):

GiCoCu - Reproduces Photoshop's handling of amp-files and Gimp's handling of color curve files in Avisynth
http://avisynth.org.ru/docs/english/externalfilters/gicocu.htm

Gradation Curves - edit the gradation curves similar to those in paint programs, with data import & export, for VirtualDub
http://members.chello.at/nagiller/vdub/index.html
and development thread
http://forum.doom9.org/showthread.php?t=133191

Using VirtualDub plug-ins in AviSynth (last post)
http://forum.doom9.org/showthread.php?p=1018607

How to achieve "film look" on PAL DV source - informative & insightful on what is nearly the same as our color correctings (with refs to above plug-ins)
http://forum.doom9.org/showthread.php?t=87791

g-force said:

Here's what you get by setting the blue output from 255 to 235, and overall gamma of 1.2

SmoothCurve( "0-0;128-158;255-255", "0-0;20-0;128-128;255-235", "0-62;50-67;100-102;128-125;255-252", dither=100, show=false )
You will need LaTo's plugin Smoothadjust, which you can get here:

I believe there are 2 misguided approaches here. First is spectrum clipping , which should never be used (while data is present) because it removes that precious information. Always slide the spectrum to where you want it, for example by adjusting gamma (but gamma itself is too specialized for this application). Second is correction should be accomplished in RGB only. Other approaches (like with SmoothCurve() using YUY2, YV12, et al, colorspaces) are not intuitive and make unintended alterations in other parts of the spectrum.

For global correction, manipulate the spectrum in RGB colorspace by setting what should be picture greys to R=G=B greys. This technique is often called "white balance". Use a "curves" function (most paint programs have one, as in my example below) where one may set the "in" and "out" of any number of arbitrary points and the program re-curves the spectrum to fit. For this proof of concept, I selected test points at low=100 and high=200 (low, medium, and high points, equally spaced across the 0-255 range, would produce an even better result):

I used "the captured princess" shot as my representative sample because it had "without doubt" pure whites (white storm trooper armor under white illumination) from dark to bright -- the posted DVD originals are on the left; the resultant adjustments are on the right. Overall, it fixes everything quite well. However, due to a whole assortment of reasons, a single adjustment may not hold true throughout and exception-adjusts should be made on a shot-by-shot basis where needed.

For this proof-of-concept, the base line (unaltered) is Blue and adjustments for true greys were made to Red and Green, as shown. Points selected for testing were Blue at roughly "100" (shadow) and "200" (hi-light) on what should be truly white objects. This is science, so use an "eye-dropper tool" and get real numbers. Then adjust the corresponding Red and Green to also equal 100 and 200 on those points. In this case:
* Red_low_input = 100 ... Red_low_output = 110
* Red_high_input = 200 ... Red_high_output = 220
* Green_low_input = 100 ... Green_low_output = 110
* Green_high_input = 200 ... Green_high_output = 215
and the rest of the color spectrum takes care of itself.

Use an Avisynth filter to do this as would a paint program (I vaguely remember seeing one, some time ago, somewhere). Otherwise, you must manually extract the 0-255 point adjusts from on your paint program curve (tedious) and otherwise apply them in Avisynth.

DuracellEnergizer said:

I like your avatar.

You do? Let me quick check! ... Oh, I thought my movie was showing again.
;)

Harmy said:

I'd like to analyse what it it is that makes STAR WARS such a great cultural phenomenon and underline its importance as an American cultural landmark.

It's a little absurd that they want us to show critical thinking in the work but at the same time we have to back every single idea with a citation, so I need sources, lots of them.

You can't know Star Wars without knowing George Lucas. If you don't know that George Lucas is a warmed-over U.S. '60's radical hippie who choose propaganda (with the art-house contemporaries he hung around, like Francis Ford Coppola) rather than "the streets", and never went underground as did the revolutionaries when "the streets" failed (and now have resurfaced inside American government & institutions to further their communism "revolution" as subversives), then you must read:

http://althouse.blogspot.com/2012/01/im-60s-west-coast-liberal-radical-artsy.html
"January 18, 2012 - 'I’m a ’60s, West Coast, liberal, radical, artsy, dyed-in-the-wool 99 percenter before there was such a thing.' Said George Lucas, who has $3.2 billion."

http://pjmedia.com/lifestyle/2012/02/10/george-lucas-confirms-it-the-star-wars-we-loved-never-existed/
"George Lucas Confirms It: The Star Wars We Loved Never Existed - Now everything is starting to come into clarity. Today is a bit like the day we learn that Santa Claus is your parents, socialism stops working when rich people’s money runs out, and a BA qualifies you for a $10 entry-level job that you could’ve gotten just out of high school."

http://mightygodking.com/index.php/2012/03/05/the-politics-of-star-wars/
"The Politics of Star Wars - ... It was the idea that we had a President [yesterday's Nixon, not today's Obama] who genuinely saw himself as not subject to the rule of law. “Imperialist Presidency” gets bandied around a lot by both sides of the political spectrum, but everyone was worried that Nixon was setting one up. And what did Lucas write against this backdrop? He wrote a story about a democracy that had become an Empire, with a ruler who (all together now) “dissolved the council permanently”."

"The truth is out there. You just have to know where to look." -The X-Files

negative1 said:

720p link: http://img269.imageshack.us/img269/1/swr3mos0643.jpg

Very cool! Your actual frame (16mm, 35mm, 70mm)? Well, on further review, I guess I really can't call it an "ink spot" (sorry, ILM, my bad) if this project's "very good, high quality accurate sources" prove not as accurate as an official DVD grab (left-film, right-DVD):

It seems the DVD grab does show some texture/color in keeping with the ground's, after all. Any instances of areas going very dark elsewhere? Perhaps the project might like to assess this.

BTW, H, no offensive soundedness noticed. :)

Page 141+? Funny! It was looking so good I guess I was expecting "Despecialized" to be a "George Goes Sane" version. ("It's been a kind of urban legend that Star Wars was mostly printed on Eastman stock so that George could count on them to eventually fade badly and keep collectors from having good copies." -- speak of The Dark Side!) Must've been traumatic XØ flashback. :D

While I was back there (p.140), I noticed the Adobe Premiere Pro screenshot. Looks sweet, but is it just a pretty picture -- as troublesome to use as the all preivous versions (inaccurate settings, false previews, half-baked filters, etc.)?

Sorry to read about the YT situation. ("When the people fear the government, there is tyranny.")

Wow, excellent project ... and that's just from seeing your YouTube video samples! Version 2 should look great as well (hope you'll do more YT samples).

This may be a little late (and of course your sample are old), but I found a couple of things that could use fixing (?) OR at least to keep in mind for the new version.

The landspeeder shot on the way in Mos Eisley Spaceport is "despecialized" but the ink spot ILM made for a shadow screams "lame". As you've reworked that entire shot, how about fixing the shadow, too?

The top section is the DVD release of the laserdisc (thanks, George!). Ink spot -- no color, black crushed, and just doesn't look like a shadow.
Your "despecialized" is the center section. Looks great ... except for that ink spot. For this proof of concept, I used the ink spot shadow area to mask and darken the same area from the clean beginning of the shot, and overlayed that result onto this frame. This allows the now darkened ground texture to show through like an actual shadow. To take this fix only one step further, you might like to adjust the contrast on the speeder to better match the surrounding scenery (darks, lights, saturation).

Also, the beginning 8 frames of the "move along" shot was jarring and made me replay it to notice a number of areas that hopefully will be addressed:

frame 1 - garbage by helmet of center trooper; frame squashed
frame 2 - same frame 1 w/o garbage or squash
frames 3 & 4 - same frame
frames 5 & 6 - same frame
frame 7 - OK
frame 8 - frames 7 & 8 superimposed (looks blurred)

Thereafter, it looks and moves fine.

Looks like another great project! I went off to your website for your screen shots and thought to make a few observations.

I would recommend, if you are able to do it, to preview a snapshot of your captures to be sure you are not "crushing the blacks" or "blowing out the whites". Using Leia's slave-girl shot for my proof of concept, the screen shot looks fine until it is compared with the released movie (I'm using the 2006 DVD release; the 1983 release, as an extra on that same DVD, was darker and muddy).

Here's the 2006 DVD anamorphic frame (720x480):

It is fairly dark, but has strong color and proper highlights. A good shot. Interestingly, when your 16mm capture is resized for DVD anamorphic, it contains most of the original picture (bordered in green):

But when it is put in place, you can see it is way too bright:

Here's the problem -- you've blown out the whites on your capture. The grabbed frame in a paint program shows the spectrum has no darks and is off the top in whites -- too bright. I had to do some fancy settings to get it roughly down to where it should've been AND had reduced the color saturation slightly (because this process is error-prone and leaves the colors too strong):

Anyway, once resized and corrected, it looks pretty good over the DVD shot:

Here's how it would look with the DVD background removed. (Will you again do some fancy edging? At least, I expect you'll trim away edge garbage that I left in here for this proof of concept.) :

When viewed with the anamorphic flag stretching it to widescreen aspect ratio, it looks great:

Even when compared to the DVD:

Of course, if the capture settings are better to begin with, it won't need the kind of fixing I applied (with the inherent distortions in that process).

msycamore said:

What bothers me in that scene are the extreme amount of haloing/edge enhancement and video dropouts.

With the source already this badly halo'ed, you must de-halo, especially before applying any sharpening, to get decent results. To demonstrate:

I max'ed the settings on "DeHalo_alpha" (rx=3.0, ry=3.0, lowsens=0, highsens=0), a script I copied somewhere -- DeHalo_alpha(clip clp, float "rx", float "ry", float "darkstr", float "brightstr", float "lowsens", float "highsens", float "ss") -- but couldn't make all the halo-ing all disappear. Maybe there's a better version? It softens the picture a little but following up with "LimitedSharpen" (made it pretty strong here, too -- strength=600, soft=true, overshoot=0, exborder=4) nicely brings the crispness back ... for a better overall result. (Of course, you would tweak the settings for better balance than I did for this proof-of-concept.)

The ScanTips website has a nice hands-on tutorial Color Correction of Faded slides . I used a slightly modified approach (he clipped the high and low ends of the RGB levels to achieve his effect; as I don't believe in deliberately trashing precious remaining information, I used gamma adjustment to achieve the same effect on all the remaining data).

To demonstrate, I grabbed a DVD frame from 2001: A Space Odyssey

to eye-ball adjustments of a faded 35mm frame.

From ScanTips' best approach, on the last page of his article (but with me using gamma), I quickly adjusted RED one way and GREEN & BLUE the opposite way.

All the while comparing the results to the DVD frame, a little back & forth fine tuning made the restored color almost magically appear. Next needed was minor adjustments of brightness & contrast

and hue & saturation

and I was finished!

(Direct RGB-video adjustment may not be a proper method for color correcting CMY-film fade. RGB seems to get it close, but RGB settings mix colors across CMY boundaries, which one would think to cause wrong coloring in those areas. It may be better to directly adjust Cyan, Magenta, and Yellow [CMY is a sub-set of RGB -- with C,M,Y being negatives of R,G,B, repectively], but I didn't try that for this proof-of-concept.)

msycamore said:

Here's a comparison with the fine transfer Jaitea provided, this will be a much better alternative than splicing in DVD footage. I'll see if I can apply some sharpening without introducing halos ...

There is an Avisynth sharpener designed to do just that -- LimitedSharpen. Halos are produced by sloppy sharpening algorithms creating boundaries instead of enhancing them:

LimitedSharpen does it the right way, which means sharpening without haloing:

See the Doom9 forum thread "Here is LimitedSharpen()" http://forum.doom9.org/showthread.php?threadid=84196 . It's 30-something pages so just review the several first and last pages for where it started and where it is today (fixes, etc.).

More information here: Avisynth - External Filters http://avisynth.org/mediawiki/External_filters

and here: Avisynth - LimitedSharpen http://avisynth.org/mediawiki/LimitedSharpen .