The Matrix - HDTVrip "Restoration" (* unfinished project *)

This discussion started in Andrea’s thread: http://originaltrilogy.com/forum/topic.cfm/The-Matrix-spoRv-BD-25-RELEASED/topic/15832/

The goal here is to take the RuTracker (open matte, original color-timing) 1080i HDTVrip of The Matrix and make it watchable.

It suffers from the following issues:

Manual IVTC Required (Pulldown Glitches)
3 Frames Missing
Horizontal Aliasing
Geometry Error
Some Macroblocking
Recording Stops During Credits

1. I fixed the weird pulldown issues by manually IVTCing and using DGIndex to fix the field-order transition errors. I verified that it's fixed by running an IsCombedTIVTC analysis pass and examining the (minimal) output.

2. I found the 3 missing frames. They were the last 3 frames before the shot where Neo enters the room to see The Oracle. I simply froze the last frame in that shot 3 times to keep it synchronized with the DVD. I will verify that the two are truly in-sync with my DumpShotChanges Avisynth function soon.

3. I fixed the horizontal aliasing by doing an nnedi2_rpow2(2, qual=3) followed by a ResampleHQ for downscaling to the final resolution (see #4).

4. You_Too helped me determine the correct geometry by manually registering some frames on top of the DVD (at 853x480, of course). We determined that the correct DAR should be ~1.75:1. The final "active" resolution will therefore be 1262x720.

5. I intend to mask the macroblocking by using a 120-frame Kodak 2383 35mm grain plate generated by poita. He says that this is likely to be the same film stock used for the release prints.

6. I intend to replace the end credits with upscaled NTSC DVD footage. Yes, it's letterboxed, but what more can I do?

AntcuFaalb said:

I found the 3 missing frames. They were the last 3 frames before the shot where Neo enters the room to see The Oracle. I simply froze the last frame in that shot 3 times to keep it synchronized with the DVD.

Are you sure that doesn't create a visual artifact (I know if's not always visible depending on where it is)?  Would it possibly be better to insert spoRv frames instead, or otherwise color-matched BD frames?

CatBus said:

AntcuFaalb said:

I found the 3 missing frames. They were the last 3 frames before the shot where Neo enters the room to see The Oracle. I simply froze the last frame in that shot 3 times to keep it synchronized with the DVD.

Are you sure that doesn't create a visual artifact (I know if's not always visible depending on where it is)?  Would it possibly be better to insert spoRv frames instead, or otherwise color-matched BD frames?

I tried playing it at full-speed and I couldn't see anything. Furthermore, there's little-to-no visible motion in those 3 frames in the DVD (it has the "Spoon Boy" in it).

If you want to see the frames I'm talking about, bust out your copy of the DVD and look at the 3 frames just before Neo goes through the beads to see The Oracle.

Regarding #5: Yes, the grain is too heavy there, but it was just an experiment, as I hadn't yet received the Kodak 2383 35mm grain plate.

There's something going on in the frame 124331... and I confirm that the final three frames of the spoon boy shot are missing, and replacing them with copies of the last frame doesn't show up, as the previous frames are almost identical.

Now I'm working to improve my own ColourMatch script, instead using the HDTV for replacing the shots with artifacts; so, AntcuFaalb, the duty of restoring the HDTV version is all yours! (^^,)

By the way, it will be easy to use all the [spoRv] soundtracks, plus the improved versions of the subtitles done by CatBus; just put the HDTV in sync with the [spoRv] - or the BD, as they have the same frame reference - and that's it.

_,,,^..^,,,_ said:

...plus the improved versions of the subtitles done by CatBus...

With that aspect ratio, my adjustments are more likely to make things look worse than better.  1.75:1 (HDTV) goes all the way to the top and bottom of a 16:9 frame.  2.35:1 (BD) has significant black bars.  So when I moved the subtitles just inside the 2.35:1 image, I pushed them ridiculously far into a 1.75:1 image.  The subtitles will be right over all the action.  I'd recommend using the straight-from-BD subtitles for this one.  You could even afford to move them out toward the edge a bit.

I have to say I'm really impressed that this grain filter of yours masks the macroblocking so well.  It makes me wonder if that's why edge enhancement was so common on early DVD releases.

Now I'm working to improve my own ColourMatch script, instead using the HDTV for replacing the shots with artifacts; so, AntcuFaalb, the duty of restoring the HDTV version is all yours! (^^,)

Now I'm working to improve my own ColourMatch script, instead using the HDTV for replacing the shots with artifacts; so, AntcuFaalb, the duty of restoring the HDTV version is all yours! (^^,)

...ehr, this is supposed to be my post (_,,,^..^,,,_) - infact, I POSTED it... but the system decided to call me "jimmi325" instead... admins, could you fix it? Thanks!

Spaced Ranger said:

AntcuFaalb said:

5. I intend to mask the macroblocking by using a 120-frame Kodak 2383 35mm grain plate generated by poita. He says that this is likely to be the same film stock used for the release prints.

Have you tried any Avisynth filters for this? I read about one (or two) some time ago, but never had occasion to try it. A quick search brought up this (and there may be others of similar or superior effect):

Doom9 forum - Filter to remove macroblocks (from 2006)
http://forum.doom9.org/showthread.php?t=108511
Q: There's too much macroblocks in the source, those 16x16 blocks is
very visible and I would like to remove it ...
A: I'd recommend you to use DeBlock_QED_MT2() and if that doesn't remove all the blocking try this script: ...

Avisynth - Blockbuster
http://avisynth.org.ru/docs/english/externalfilters/blockbuster.htm
Blockbuster is an Avisynth filter designed to reduce or eliminate DCT blocks from an enocode. ... (Incidentally, DCT stands for "discrete cosine transform", and is one of the techniques MPEG uses to do its compression. They differ from macroblocks, which are most noticible in high-motion areas when the bitrate is insufficient to describe the motion accurately, and also from mosquito noise, which is an artifact that tends to appear around edges and areas of high contast.)

I have experimented with many such filters, including the deblocking algorithm that comes with DGDecode (MPEG2Source).

The issues I have with them are (1) they definitely blur-out some detail, even in the best case and (2) the source in this case starts to look really really ugly and waxy.

I vote for the grain plate!

But, what about a 1080p version? Too bad quality Vs 720p?

_,,,^..^,,,_ said:

I vote for the grain plate!

But, what about a 1080p version? Too bad quality Vs 720p?

There just isn't enough detail to justify full 1080p. Too much was lost during the original MPEG2 encode.

Using the 35mm grain plate (obviously) boosts the contrast. I intend to pull it back down and use a combination of AverageLuma (in ScriptClip) and "analyze=true" (in RGBAdjust) to verify that my tweak to pull down the boosted contrast results in something that's as close as possible to the HDTVrip source as possible.

I'll post my script-in-progress tonight.

Is anyone interested in helping out with further testing?

drngr said:

Certainly.

OK, cool!

I'm mostly working on being 100% sure that the shot-changes are aligned. Everything seems perfect so far.

I can send you a rough copy of my Avisynth script tonight, if you'd like.

Correction: Further registration tests have led me to choose exactly 1.75:1 as the DAR.

Also, FWIW, excluding the unmatted parts, I'm not seeing much more detail in this when compared side-by-side by the DVD.

drngr said:

Certainly.

Let me know when you're available and I'll send you the script.