Image registration on video frames (experimental)

I am experimenting with a method for registering frames from different video versions of Star Wars. This was attepted before by the x0 project, and Mike V is doing something similar with his Legacy project. I guess with the Despecialized Edition and other upcomming HD projects, doing this in SD is not very relevant anymore, but I'm still interested in it and so might be others.

What I'm trying to do is saving successive frames from two different versions of the movie, and then register frames from one to the corresponding frames in the other version using the command line tool Elastix. Another command line program, ImageMagick is used for some clean-up work. The whole thing is glued together by a primitive Common Lisp program (or more precisely, a set of functions) and a shell script. I'm doing this on Mac OS X by the way. Adapting the systom for other Unix clones sould be trivial. It's also possible to adapt it for Windows, and I may do so one day.

There are a couple of free tools capable of doing image registration, for example the TurboReg plugin for ImageJ. I had the most success with Elastix, and also it is the easiest to use in a batch-like method.

The two sources I'm using is msycamore's Technidisc and Harmy's Despecialized Edition 2.0. The Technidisc is my absolute favourite version of Star Wars on video, and a great LD rip. IMO it has the best color palette and generally the best distribution of lightness levels, even though at some places it is inconsistent. Aliasing is less prominent than in the GOUT (thanks to the overall softness of the image). Light sources don't bloom (no fat lightsaber blades). The plan is to borrow detail from the DeEd2 and try to fix some of the problems in the Technidisc.

Currently the program is capable of going throu a queue of frames automatically. It will also fix undershot pixels (near-black pixels that, after the B-spline transformation, turn into negative and after conversation end up white-ish). As you can see on the samples below (especially on the fourth sample, on C-3PO-s highlights), overshot pixels can also happen. Need to fix those too. Using linear interpolation doesn't cause any overshot pixels, and the quality of registration doesn't suffer either, so problem solved. :)

Another problem I want to point out can be seen on the last sample. It's partially a registration error (upper left corner, the planet and surrounding stars), and partially the fact that the fighters were composited a bit differently in the DeEd (see the Y-Wings).

To do:

• Fix overshot pixels
• Tweak registration parameters for more precision in space-type scenes
• Also tweak more for speed (on a 2.2 GHz Core i3 notebook it takes 2-3 minutes to register one frame)
• Find a better method for extracting detail from the registered images - current method alters the luminance slightly
• Use the transformation data from one frame as starting point for subsequent frames and reduce the number of iterations during registration on them, since the deformations will probably be similar.

Could this be used for frame count sync for audio to video?

Very interesting! Every attempt to create a fully working system to use different versions together to get more detail is great.

Jetrell Fo: Do you mean to detect differences in frame count between different versions? I think doing that "by hand" would be much faster.

You_Too: The final result of this experiment would only be a "Technidisc Deluxe": adding detail from a HD source to the Technidisc. I'm not sure if accumulating detail from multiple SD sources would give us any more than we have with the GOUT now, ironically. The GOUT itself could also be added to the mix of course, if we could find a way to cancel out any DVNR-plagued data.

New samples:

I re-registered this frame from Moth3r's PAL DVD, Alignment is much better. I also did some refinements in the composition phase (when the detail is added to the Technidisc frame) so it doesn't look so crude now as with some of the first batch of samples did.

Two new frames:

I'm working on a video sample now.

poita said:

That is interesting work, I have some registration routines I can share with you going forwards with this.

 

Are you attempting a super-resolution output from the result?

Two different telecines will capture details that is missed on each on its own.

Fine detail data may land on a scanline on one transfer and not on the other, and vice-versa. It should be possible to recover both lots of data and end up with a better capture than either of the originals at a higher resolution.

 

Registering many versions and multiplexing them in a super-resolution manner would also be possible. I went with the DeEd because it has all the details you'll ever have with this movie, and you have to do only one registration pass to get it. :)

I'll have to do a bit of a clean-up on the code, and after that I will upload it somewhere for anyone who may want to try it.

Video sample: link.

I uploaded the program itself if anyone wants to look at it. For instructions see the readme.txt. Link.

_,,,^..^,,,_: The method of extracting detail from the DeEd frames after registration is similar to what's described here. I made the first sample frames in Photoshop, and the video sample is made with this Avisynth script:

tech = ImageSource(fix\tech/04d.png", start=20, fps=23.976).Trim(0,609)

reg = ImageSource("output\test%04d.png", start=6, fps=23.976).Trim(0,609)

 

techblur = tech.GCBlur().GCBlur()

 

reg_ovl = reg.Invert().GCBlur().GCBlur()

reg_detail = Overlay(reg, reg_ovl, opaity=0.5, mode="Blend")

 

hardlight = Overlay(techblur, reg_detail, mode="HardLight")

softlight = Overlay(techblur, reg_detail, mode="SoftLight")

 

luma = Overlay(softlight, hardlight, opacity=0.75)

result = Overlay(tech.ConvertToYV12(), luma.ConvertToYV12(), mode="Luma")

 

return StackVertical(tech.ConvertToYV12(), result)

 

 

################

 

function GCBlur(clip input)

{

return input.ConvertToRGB32().GeneralConvolution(0, "

10 10 10 10 10

10 10 10 10 10

10 10 16 10 10

10 10 10 10 10

10 10 10 10 10", 256, False)

}

The GeneralConvolution function in Avisynth seems to serve this method much better than Photoshop's blur filter. Looking at the samples in my first post now, they don't look convincing at all.

msycamore: I know you are a perfectionist so I hope you don't take it as an insult when I say: I think you did a very god job with what you had. :) Seriously, your DVD shows more of what SW could or should be color-wise than any other video version, so that's what makes it valuable. The only other transfer of SW that has validity in it's color palette IMO is the early PNS with the soft skin tones (after lowering the gamma), but that is a re-interpretation of the SW look for video, while the Tech is closer to the real thing. Thanks for releasing it on DVD!

Sorry, need to work on my acronyms. I meant Pan & Scan.