DrDre

towne32 said:

Someone asked this before and I'm not sure if there was an answer. But, if going the route of a single correction applied to an entire film, is there a way to simply make an LUT out of the original and modified frame? 

 In essence, that's the output of the algorithm (for the single color space model). For 8 bit and 16 bit images this can be done, exactly. However, for the exr images this is not possible, as there are an infinite number of possible color combinations, which is why it is a HDR image format. 

There have been many discussions about the color grading of the recently released Raiders of the Lost bluray. Although many liked the new colors, others complained about the orange and teal look of this classic film, and prefer the color grading used for the dvd’s (and used in a number of HD broadcasts), released in 2003. However, in my opinion neither is actually representative of the original 1981 color timing. For example, both versions have a very noticeable red shift in the bar fight scene, that is not in the release and re release trailers, or in the 1980s home video releases. This project is aimed at creating a version of Raiders of the Lost Ark closer to the original theatrical colors. Although an original unfaded 1981 35 mm print would be the ideal reference, sadly it is not available. As such I will use the 1984 laserdisc release as a reference for a shot by shot regrading:

Stay tuned for more information…

================================================================================================

Original start of the thread:

I used this 35 mm frame as a reference:

Calibrate a color correction model to match the bluray to this frame.

Bluray:

Bluray matched to 35 mm frame:

Here’s how the color correction model would color grade a number of other frames (bluray top, correction bottom):

Kurosawa10 said:

DrDre said:

Here are the predictions for some of the other frames:

So, in principle you could regrade the entire film, based on the single frame with this method, and it would look pretty awesome. I'm sure these predictions much closer to the 35 mm colors than either the bluray or WOWOW. Since this method is much faster, it would take about 2.5 days to do the entire film. Oh, and those nazi flags are definitely red in this regrade.

 If one were to regrade the entire film, what would be the best way to do it? Export the whole film into images via e.g. virtualdub and load them using your program? It would certainly be awesome to watch Raiders with those colours.

 Since it won't take much time, I'm considering doing this regrade myself. I'll start a new thread on the subject.

However, if you have enough references available, the multi color space match/predictions within a shot are the way to go:

Bluray:

Bluray matched to Tech IB with single color space method:

Bluray matched to Tech IB with multi color space method

It has to be said that the Star Wars bluray is notoriously difficult to regrade. In other cases the single color space method works really well, like for the Raiders 35 mm frame, and as such it's predictions are more reliable:

35 mm:

Bluray matched to 35 mm with single color space method:

Here are the predictions for some of the other frames:

So, in principle you could regrade the entire film, based on the single frame with this method, and it would look pretty awesome. I'm sure these predictions much closer to the 35 mm colors than either the bluray or WOWOW. Since this method is much faster, it would take about 2.5 days to do the entire film. Oh, and those nazi flags are definitely red in this regrade.

nightstalkerpoet said:

:)

Here is that last shot (predictive from a different 35mm frame match) with slightly less contrast, compared to DrDre's Wowow to Blu match.

http://screenshotcomparison.com/comparison/143259

 Great results! Something similar actually can also be done in the GUI in principle, if you change one of the parameters of the model. This is based on color matching in a single color space. The matching isn't that good, but predictions in other scenes tend to be more robust. I have implemented this in an update of the GUI. There are two options for model selection: multi color space model (default), and single color space model. The latter is also much faster, but as I said if you're looking for exact matching, you need the first model. If anyone is interested, please send me a PM. 

Here's an example of a prediction for the single color space method using the 35 mm frame I posted earlier as a reference:

Bluray:

35 mm frame:

Prediction from single color space model:

Kurosawa10 said:

I just created a correction model using the WOWOW referance frame that was posted earlier. (This one: http://screenshotcomparison.com/images/1442247198_4817509236.png )

I tried using the correction model on another frame from Raiders, but I can't say whether or not the corrected frame is better looking than the original bluray frame. The result is either way quite amazing.

http://screenshotcomparison.com/comparison/143209

 As you can see the early parts of the film were color timed a little different than the later parts. Using a frame as a reference from the same shot, results in a much better prediction:

Bluray:

WOWOW:

Bluray matched directly to WOWOW:

Using this color correction model for the prediction leads to:

Bluray:

WOWOW:

Predicted color correction using color correction model calibrated on the above frame:

Zyrother said:

In the end here, the original goal of determining whether Super Resolution was a viable option in up scaling the GOUT was a tremendous success. Comparing the original screenshots to the chosen V11 script is startling. 

It was a community effort and everyone did excellent work! :)

 Amen! :-)

towne32 said:

It would be nice with the kind of shot change detection that exists in software like pfclean. Have the whole reel/film laid out into shots and choose the representative source shot/frame and apply it to as many target shots as you like until your whole sequence is covered.

 Yes, that would certainly be ideal... 

schorman13 said:

Dr. Dre, I wanted to say thanks for all this work.  This really is an amazing and useful tool you've shared.  

As far as whole films go, I would think the tool would have to be automated somehow.  It's one thing for it to take 20 seconds per frame, it's another to have to manually load and process the ~200,000 frames in a 2 hour film.  

Ideally you could just import 2 uncompressed avi files and output a third, rather than dealing with giant lists of png or bmp files.

I'm sure this would require much more development.  Frankly, I'm just happy with what this version can do.  

If there's a way to output a color correction matrix file for each frame, some other avisynth plugin (ColourLike?) might be able to do the color correction separately.

 Sadly, that is not possible, as far as I know. You still would have to know which references you would like to use, which frames to correct based on these references, etc. Color correction still requires a lot of work, and a human eye to check for anomalies, even with the help of software tools like this. 

Swazzy said:

What would be the easiest way to implement this algorithm to an enitre film/shot, say through avisynth?

 Use Virtualdub to export the frames (or a selection of frames) to image files. Correct them using the GUI. Import the frames in Avisynth again (the names of the corrected files are the same, except in another directory, so this is easy). Add sound, and you're done. 

NeverarGreat said:

Okay, I've done my first color test with the software and the results are quite accurate. FYI though, I had to reverse the order of steps 1 and 2 to get it to work. The biggest problem is that the Blu-ray has such compressed gradients that the color is flattened, and no algorithm can fix that.

Correct me if I'm wrong, but as of now this program will only accept separate images, so it's rather awkward to use this for an entire shot, much less a whole scene. So for me it's back to correcting entirely by hand, at least for now ;)

Theorizing here for a minute - Say you match the color of the Star Wars Blu-ray to a source like a 35mm scan or the GOUT, and then the program compares the final result for each frame to that reference source. Any colors that are matched 100% remain, but any colors that don't match, say, because of missing gradients, are then added to the Blu-ray through some sort of color blend mode (which I've done to the Blu-ray using GOUT color in certain places). This would require first registering each frame. Thoughts?

JEDIT - Another idea, instead of cropping the reference and test images, perhaps the program could employ a system of target points that the user can place on the frame and essentially tell the program to correct to those, rather than the entire frame. So details that are important, like C-3PO's gold color, or the skintones, will be weighted more heavily in the correction than other parts.

Regardless, and echoing what others have said, it's an exciting proof of concept and I look forward to seeing where you go with it!

 Actually, once you've build a model, you can select any number of frames you like for correction. Depending on the number of shots, you could accurately process 50-100 frames with one model. As I've shown it is very accurate within shots. 

The model doesn't directly compare pixel colors, it compares color distributions in many different color spaces. It then matches these distributions, such that the colors match. That's why you can match colors between two sources with a different resolution. It does require that the two frames represent the same ground truth, so to speak, therefore they should be cropped in the same way.

Also the information about the gold color is embedded in the other colors. Take the C3PO frame. I built a model using only a tiny bit of the frame, which includes R2D2:

I select the same part in the reference, create a model, and predict what the rest of the frame should look like:

It is an accurate representation of the corrected frame, predicted only by the color changes on R2D2, and a tiny bit of the surroundings. 

Let's get back to the discussion of regrading a source to approach some kind of reference, with only limited resources aka reference frames. Take this frame from Raiders of the Lost Ark for example.

Bluray:

35 mm frame:

We can match the bluray to the 35 mm easily:

However, if I were to ask you to regrade these two frames from the same location to look like this 35 mm frame:

Would you be able to do it? How much time would it cost?

This is what you get from the prediction, in about 20 seconds:

First of all, I think they look stunning, but more importantly, they look like frames from a 35 mm print, consistent with the reference. 

RU.08 said:

DrDre said:

Hope you enjoy the tool. Of course if you use the tool for your projects, any acknowledgements will be appreciated. The same is true for any comments, critisism or suggestions you may have. In that case write a post in this thread or send me a PM. 

Can you use command-line processing? Example:

for %f in (*.tif) do regrade "%f" -ccm "correctionmodel.txt"

 Sadly not, MATLAB is a scripting language, so within MATLAB you can use command lines, but since this is a standalone application, you can not do this anymore.

revgen said:

I decided to test it out last night on the Criterion & Arrow versions of Dress To Kill. Caps were grabbed from DVDBeaver.

Here's the Criterion Blu-Ray.

Here's the Arrow Blu-Ray.

Here's the Criterion Blu-Ray Color Matched to the Arrow Blu-Ray.

Other than a slightly pinker shade to Michael Caine's skin tone, the corrected image looks very accurate to the Arrow Blu-Ray.

 Take care to crop the frames in the same way, because it influences the accuracy of the color matching. I corrected the same frame, and the colors are an exact match:

The second situation, is when we are matching a print, in which case color inconsistencies within an aging the print may play a role. However, even for the Star Wars bluray, which probably has the most inconsistent color timing ever, accurate predictions can be done within shots.

First the model is calibrated on this frame:

Bluray:

Bluray matched directly to Tech IB:

Then I predicted the folowing frame:

Bluray matched directly to Tech IB:

Predicted correction with color correction model from the above reference frame:

The prediction is pretty much identical.

FrankT said:

Which would you say has the better colour?

 I think I generally prefer the WOWOW, because the bluray is very orange, but in this scene the bluray is looking pretty good.

Generally, the further you go away from a reference frame, the worse the prediction. However, there are a couple of situations we can consider.

Let's consider the case that you want to match two good quality sources that have been color timed in a different way. It is possible that each frame was color timed differently, however usually the color timing within a shot or even in a whole scene are very similar. 

Here's an example for the Raiders bluray and the WOWOW. First the model is calibrated on this frame:

Bluray:

WOWOW:

Bluray matched to WOWOW:

First, I predicted the corrections for in the same scene, within the same shot:

Bluray:

WOWOW:

Predicted correction with color correction model from the above reference frame:

Then I predicted the corrections for in the same scene, but not within the same shot, where the color timing differences are expected to be larger:

Bluray:

WOWOW:

Predicted correction with color correction model from the above reference frame:

The prediction is still very close. The color of the whiskey is actually more consistent with the reference in the prediction, than in the actual WOWOW frame. Certainly far better than could have been done manually without a good reference, but we should be careful using frames from a different shot.

towne32 said:

I agree with Neverar. Dre, I'm certain that it was unintentional, but your Star Wars images there are literally the exact same url. Here's the comparison, at least using the images you had a couple days ago:

http://screenshotcomparison.com/comparison/142387/picture:2

You can see that it gives the same general impression and tones. But it's lacking the specifics and looks more homogeneous overall. The shadows look a bit funky, too. If you look at shots 1 and 4, I think there's really no competition. The specific match looks like a technicolor print. The non-specific looks like it's in the ballpark, but not right. 

The predictive ability is really neat that it even works as well as it does. A great proof of concept and I'm sure there will be cases where it's useful because of limited resources being available. Now, I'm not suggesting it needs to be frame by frame, but probably shot by shot or as new elements enter a shot. Applying one ref over a wide range of frames seems like it would be risky for introducing artifacts and incorrect color.

edit: I am curious if your second Indiana example WOWOW and Match might be identical images. It seems the actual WOWOW might be missing.

I replaced the duplicate images for both the Star Wars and Indy examples. I also switched to another image hosting site, because screenshotcomparison can be frustratingly unstable at times.

You actually used the C3PO/Owen image that did not have the correct cropping, so the colors came out wrong. The one that's in the example was the correct one, posted after neverar great noticed some problems. The correct prediction is very close.

Let me eleborate a bit in the next posts...

Here's another example of prediction for the Raiders of the Lost Ark bluray, and WOWOW. First a color correction model was calibrated on this frame:

Bluray:

WOWOW:

Bluray matched to WOWOW:

Then I predicted the corrections for the following frames:

Bluray:

WOWOW:

Predicted correction with color correction model from the above reference frame:

Bluray:

WOWOW:

Predicted correction with color correction model from the above reference frame:

NeverarGreat said:

Quick comparison:

1: Blu-ray

2: Blu-ray with predictive colormatching from 16mm source

3: 35mm frame with slight color correction to remove red fading

4: Blu-ray with curves adjustment

While the colormatching algorithm works amazingly well when using identical shots, I'm not sold on its predictive power. In the above example, the predictive shot has crushed and dull highlights, and the blue in the background inexplicably turns to purple.

The model was not calibrated on a representative 35 mm frame, but on a 16 mm frame, so naturally it's not going to be able to predict the colors of a different print, and certainly not one in poor condition. However, if the source and reference quality is good, the predictions are usually very accurate in the same scene, and sometimes across scenes, as was evident from the Star Wars example:

Bluray:

Bluray matched to reference frame:

Bluray:

Bluray matched directly to Tech IB:

Predicted correction using color correction model from the above R2D2 reference frame:

Interestingly enough, the 35 mm scans provided by Harmy seem to suggest the film indeed was more contrasty than either the bluray or the WOWOW:

35 mm:

bluray corrected: 

Here are a few more iconic frames from the film adjusted by the model to match 16 mm colors:

I actually did slightly correct the reference, before building a color correction model. It is very contrasty, but I think that's part of why it has that film like look. It would never look like this on a home video, but it does recreate the atmosphere of being in a theatre. Here are a few more iconic frames from the film adjusted by the model to match 16 mm colors:

This is the result. Who says you can't make the bluray look like film? ;-):