logo Sign In

Idea: Personalized preservation possible with September 2006 OT DVD's

Author
Date
Author
Time
 (Edited)
Post link

Perhaps I’m one of the few, but I’m actually glad to receive both the 2004 and OUT discs. It opens up many new possibilities for doing preservation releases. Consider this:

  • Everyone now has access to the exact same digital master copies.

  • Everyone now has access to both movies.

  • I didn’t buy the 2004 versions before (thank you laserdiscs)

Why does this matter? Well, we didn’t get the anamorphic film transfer we wanted, so IMO Ocpmovie’s approach of using the 2004 discs and splicing in original footage still seems to be the best bet. But now that we have two digital sources, perhaps it can be done with more choice! Here are the proposed features:

  • The main feature will be selectable patches. This will be applied to the 2004 DVD to improve it. We don’t need to distribute the entire movie… just the patches. This should mean a much smaller download. I’m working on a solution to this, but it is still in the early planning stages. Many MPEG/AC3 details remain to be worked out (see below). New ideas for patches will come along in the future, but the old patches still remain and are just as usable, so work does not have to be duplicated (for example, once a patch exists so that Han shoots first, it doesn’t have to ever be redone, unless a change applies to that specific scene).

  • Basing edits on patches means customized editions, based on personal preferences. For example, if you want to see a ring around the death star explosion (or like garbage mattes, Hayden Christensen, etc), you can deselect that patch… each edit will be optional. Since we can’t all agree on which changes to keep, menus, special features, etc, I propose choices for each. If it looks like something can’t be changed, bring that up.

  • Again, we don’t have to distribute the original movies, so I hope this avoids a lot of legal problems, or failing that, at least personal moral issues.

  • I envision a Linux live CD, with everything ready to go, you just supply the September DVD’s… custom made for our preservation project. Why Linux? Tools will generally need to run from command line to be automated decently, and more of these types of programs and libraries are available for Linux… also, I’ve switched to Linux so have no desire to work on a Windows version. If a PowerPC version is desired, this should be possible, although I don’t have the hardware necessary for this. I’d be willing to work with someone to port over whatever I came up with.

  • The parts that aren’t patched will not need to be re-encoded: the quality of the '04 version is preserved.

  • Post processing options can be made available (contrast/color changes, sharpness, etc). These effects would be separate so that they could be applied to scenes independently from the patches above. Of course, these would require a complete re-encode of the video, so perhaps they are not great. I bring it up because someone mentioned the 2004 versions being too bright and clean vs the originals.

  • Other re-encoding options are possible, if really desired, such as 4:3 P & S. Also export to other computer formats: Divx, Quicktime, etc.

  • Audio: Most of the above will apply as well. For example, I personally prefer C3P0 not to tell Obi-Wan how to disable the death star (but Ocpmovie does), so audio options need to be available. Also, it would be interesting to have alternate audio tracks such as 5.1, audio commentaries, etc.

  • Have a choice between: Dual layer, 2 disc DVD5, or Dual layer -> single layer requantization (i.e. DVD9 -> DVD5). The only reason I don’t stick with Ocpmovie versions is because the single layer re-encoding of the movie makes it look awful.

  • Choice of menus and special features.

  • Add other files to the disc (PC DVD content, jacket covers, etc).

  • Finally, burn the DVD’s, write an ISO, Divx, etc

Of course there are some problems:

  • You’d have to buy the new September DVD’s. I know not everyone is excited about that… but it is obvious that this approach requires it.

  • All the great fixes of the past will have to be redone… again. But, hopefully this would be the last time!

  • Doesn’t solve the NTSC vs PAL problem. I see no way to resolve this.

  • I’m pretty happy with the Faces laserdiscs. Maybe they are good enough?

Unresolved details:

  • How are the patches stored? As movie clips, still frames, diff’s to frames from the 2004/September discs? Is there a way to distribute patches that stays legal?

  • What about subtitles? These might be quite difficult to integrate with the movie.

  • Chapters? Do we need to change the chapter stops or are the originals fine?

  • Does it matter if the program is text or gui based? It would be a lot easier to use text/curses based dialog tools (such as seen for Debian/Ubuntu Linux installers, make menuconfig, etc). Could always go GUI later if there is a demand.

  • Editing granularity, is splitting on I-Frames okay? From what I read this means that each patch would cover about 15 frames (about 0.5 seconds of video). There might be ways of working around this… I need to become more familiar with what is possible.

  • Need to find AC3 cut/join software.

  • Anything proposed that nobody cares about? No reason to unneeded extra work…

  • Is anyone interested in this project (i.e. those that would be creating patches)?

I appreciate your constructive feedback and ideas.

Thanks!

Author
Time
Post link
According to Wikipedia, "Maximum frames per GOP: 18 (NTSC) / 15 (PAL), i.e. 0.6 seconds both." So you're a tenth of a second off on your editing granularity. And outputting to Divx or Quicktime on Linux is a sin; use Xvid or H.264 in an ISO mpeg4 container. Or maybe Ogg Theora will be ready someday.

Edit: I'm stupid and overlooked the word "maximum." Ignore me.
"It's the stoned movie you don't have to be stoned for." -- Tom Shales on Star Wars
Scruffy's gonna die the way he lived.
Author
Time
Post link
OCP's Classic Edition might be done well to be remade using the September disks, since the quality will be much closer. Will be interesting to see how close they match up to see if this is viable.
Author
Time
Post link
I have had an idea not far from this, for fan-edits.

The idea was to allow fan-edits to be distributed legally as data files (".edit") containing no copyrighted material. Instead the files would have to be played on a computer using special player software which would play scenes from DVD in an order specified by the file.

The file format would be based on a zip archive.
The main file inside the archive would be a play list - a text file with ordered commands to the DVD player/s on what frames and audio tracks to play. A somewhat smart subsystem in the player would cache frames in memory (not break only on GOP boundaries). It would also cache entire sequences on the local harddrive so that users would not have to switch discs more than once.
The .edit-files would also be able to contain video sequences, but these should have to be encrypted with DVD data as decryption key (XOR algorithm = simple, superfast, yet totally secure).

Subtitles would have been stored as .srt-files. Menus would be HTML with the usual features of pictures, scripting and Flash. This would be OK for realtime playback on a computer.

There were two major problems:
1. The legal issue of breaking DVD copy protection. There are no 100% legal DVD players on Linux that I know of. There is no public DVD-API on MacOS. I don't know about Windows.

2. For realtime playback, the computer and DVD-playing subsystem has to be fast enough to decode two streams in realtime. If you circumvent DVD copy protection, do you still get hardware acceleration? Does hardware acceleration support multiple streams and cutting in the middle of a GOP?

Problem 1 still persists if you make a Live CD.
Author
Time
Post link
Originally posted by: zombie84
OCP's Classic Edition might be done well to be remade using the September disks, since the quality will be much closer. Will be interesting to see how close they match up to see if this is viable.


The quality loss that I'm referring to is encoding quality. Ocpmovie's Star Wars: Classic Edition was reencoded to fit on a single layer. The stars look like weird jpeg block artifacts and there are other encoding artifacts everywhere. The quality of the original work done is great, but ruined by squeezing it down to a single layer disc.. also, I disagree with some of the changes, as I'm sure others here do. And, the sound quality is poor (I guess it was recorded from the laserdisc?) So the edition has some problems.

I'm proposing a system that allows choice in what edits take place, what postprocessing is done, menus/special features, and how the result is output. The idea is to give more choices, not less (so thanks for mentioning those other output formats, any format that mencoder/ffmpeg are capable of outputting would be automatically available).

Those numbers (15 frames/0.5 sec) were taken from the mpgtx man page. I admit to little knowledge of film editing, but I didn't want to spend a bunch of time figuring things in detail out for a project nobody was interested in. If there is interest, I will do the research needed to make sure it turns out well. The last thing we'd want is to make a bunch of patches then find out that cutting them in caused the sound to go out of sync, glitches in playback, etc. I remember some posts by Ocpmovie where he had trial and error time resyncing the audio for Return of the Jedi. My system will have to figure it out automatically, so there are certainly some technical hurdles to overcome.

If the loss of credit/fame for a patch is a concern: there can be a comments field for each patch for describing the patch, methods used (whatever, really), there can be a standard template, and the author of the edit will be right at the top.
Author
Time
Post link
There were two major problems:
1. The legal issue of breaking DVD copy protection. There are no 100% legal DVD players on Linux that I know of. There is no public DVD-API on MacOS. I don't know about Windows.

I've thought about this. I see two options: either the user can rip the DVD's to their hard drive themselves using whichever program they have, or the disc can download DeCSS from the repository and install it automatically (it wouldn't be stored on the disc anywhere). After the patches are applied, they can either have it burn automatically, or just have the ISO (or the other options I mentioned, Divx, etc). So the Mac user wouldn't need DVD access if they ripped the discs beforehand, and then burned from the generated ISO.

2. For realtime playback, the computer and DVD-playing subsystem has to be fast enough to decode two streams in realtime. If you circumvent DVD copy protection, do you still get hardware acceleration? Does hardware acceleration support multiple streams and cutting in the middle of a GOP?


My system doesn't aim at realtime results, rather choice in edits and high quality results. The patches would be applied and output (and will probably take some time), then afterward you could burn the ISO, convert it to a computer format, etc. So you'd have a copy of your personal edit, generated by the tool. The disc would be running in a text console, so playback wouldn't be possible anyway (well I guess there's always SVGAlib). Playback isn't one of my goals for this project... that is handled better by other apps or hardware players.
Author
Time
Post link
Originally posted by: Darth Lars
The .edit-files would also be able to contain video sequences, but these should have to be encrypted with DVD data as decryption key (XOR algorithm = simple, superfast, yet totally secure).


Great idea, thanks! I missed this when I first read your post. I kept wondering how I could lock the patches to the OUT DVD, and this is the way. Only someone who actually has a copy will be able to decrypt the patches.

I've reconsidered my boot cd idea. I don't think it is really necessary, and will cause problems (for example, networking setup, USB, lack of NTFS write support, etc). We can hopefully port to Cygwin for Windows users. I have no way to personally port to OS X, but we can deal with that later when the time comes.

I'm glad to see some interest. I've created a very poor (at this point) wiki page, which will hopefully improve as we map things out. I am pretty tired now so I'll have to work on it more later:

http://kidsquid.com/dokuwiki/doku.php?id=september.personalized.original.trilogy
Author
Time
Post link
The Linux mpeg-2 editing tools I've found so far aren't capable of doing what we need (for example, they get confused by the episode 3 scroll). I've located the ISO13818 (documents describing the MPEG2 standard), and I'm slowly writing my own decoder, with the goal of understanding MPEG-2. This is going to take a while: MPEG-2 is extremely complicated. But, the documentation is well written and hopefully the knowledge I gain will make advanced editing possible, beyond what we could hope for with standard programs.

I envision a customized film editing program. This would solve several problems at once:
  • Eases editing: Those creating patches won't have to worry about details like numerical offsets, lengths, etc.
  • Cut, join, etc are now a lot easier, because the video/audio can be shown and matched.
  • Edit history can be recorded and "played back". This will hopefully keep us legal? What this means is that rather than store a frame or video, it just stores a list of instructions of things to do (the edit history), so applying a patch plays back the history and the film becomes edited as if the original did it themselves. So, no portion of either video needs to be saved in patches. This should keep the patches small.
  • Should make it possible to do special edits where the video isn't re-encoded, or only the modified portions of the sequence are re-encoded (subject to the restrictions of MPEG-2).
  • The editor would understand the MPEG-2/DVD format so it won't be confused.
  • Editing would be done at the GOP level, so the whole sequence (and changes) could be seen at once.

This may be somewhat unlike a normal paint program. Only certain frames might be editable and editing those frames will affect the others. And, certain effects might be possible while others may not (it all depends on how the compression is done, which I have not even begun to explore). I'll be able to tell more as I learn more. Additionally, the tool set will probably be quite limited compared to common paint/sound editing programs, but that's okay, because we don't most of that stuff anyways. Also, we can go beyond and provide specialized tools as needed.

Anyhow, the first goal is to understand the spec better, then I can see what to expect from there. At the very least, I need to make cut/join tools that work.
Author
Time
Post link
Sounds a lot more complicated than what I was thinking. I thought someone would just take some clips from the September DVDs, make them anamorphic, upload them, and we could drop them into whatever editing program and make new versions ourselves.
Author
Time
Post link
Yeah, but then we couldn't fix problems such as garbage mattes. When the film has to be re-encoded it looks terrible. So, I'm trying to avoid that if I can. Who knows, it may not work out. Still too early to tell.
Author
Time
Post link
Interesting idea, although probably won't work as well as we'd like.

I like the idea of posting the little clips of OOT footage in anamorphic, for us to drop in ourselves. I'll be doing that anyway.
I hope this idea works though, it sounds very interesting.

I know I’ve made some very poor decisions recently.

Author
Time
Post link
Sounds like you're putting a lot of work into this, calamari, which is great. I think you might run into problems matching the video quality of the two sets, though. As you know, the SE set has been extensively "restored" which has resulted in an image that wouldn't match even a true anamorphic transfer of the OUT. If you want to avoid the kind of mismatch problems ocp had, you're going to have to run the OUT sections longer than just patches (i.e. whole scenes). And even with that, you'll have to do extensive color/gamma correction on the SE scenes to get them close to the OUT look, which will mean re-encoding. Add to that the re-encoding required for the OUT scenes due to the fanamorphization and I wonder if the end result will be better than just watching the OUT disc. Don't get me wrong, I wish you every success and I think your approach is a good one in principal, which could be useful far beyond its initial inspiration. I look forward to seeing the results.
Author
Time
Post link
THX,

You bring up some very good points. Since I haven't done film editing, it is very helpful to hear what kinds of problems arise and how they are currently being fixed. Thanks!

One step of MPEG-2 encoding (and I guess JPEG as well?) is a DCT, which results in frequency information (and some of that information thrown away, for the compression). What I'd like to try when I get there, is to manipulate that frequency information and see how it affects the image. If I can cause a color change, then we might be in business. If that doesn't work (and of course, it's pretty unlikely to work) I see two options: 1) Match the OUT scenes to the SE (results in less re-encoding, since the OUT scenes will have to be re-encoded anyways), 2) Re-encode afterwards, as you say. I'm also hoping that I can pull off the idea of a partial frame re-encode. I'm still decoding program headers at this point, so I'm pretty far off from decoding frames.

I also wonder if the end result will be better than the OUT disc. But, "I have to try". If none of the MPEG tricks end up working, at least everyone can choose the patches they want, which I hope is a welcome benefit.
Author
Time
Post link
Matching the OUT scenes to the SE will indeed require less re-encoding but there are two drawbacks to that plan:
1) the OUT segments will start at a lower resolution; they will be color-corrected, scaled and then re-encoded - while making the color and delivery resolution match, this will make the OUT video quality even lower, while retaining the high video quality of the SE, so the two sources will simply mismatch in a different way;
2) the purpose of the exercise is presumably to recreate the OUT as accurately as possible with available materials, but the SE('04) colors are totally inaccurate, so matching them doesn't make sense.
Some forum members who might be of assistance to you: tellan, DarthEditous, ADigitalMan, mverta.
Author
Time
Post link
I have not actually seen the 2004 SE's, just the 1997 SE's, and even those only 1-2 times each. I don't remember any color problems, in the theatre I was too distracted by super-mario sarlaacs and explosion rings. A friend of mine has the 2004's, I should borrow them just to see what I think... my icon shows why I could never fully enjoy the 2004 SE's .

Now that I think of it, I have seen parts of the 2004 SE's from the Episode 3 soundtrack bonus DVD... but often clips like that are of lower quality.

All: I've updated the wiki.. it should have pretty much everything discussed so far. If anything is missing or you'd like to change things, I encourage you to do so: SPOT wiki
Author
Time
Post link
Originally posted by: calamari
Doesn't solve the NTSC vs PAL problem. I see no way to resolve this.
Seperate patches. But at the end of the day both should have exactly the same *frame* count, so if you write a generic patch to cut at frame 11850 then that should be the same frame on either a Pal or Ntsc disc. In theory! Don't stone me if I'm wrong!!

Sounds interesting anyway.
Some were not blessed with brains.
<blockquote>Originally posted by: BadAssKeith

You are passing up on a great opportunity to makes lots of money,
make Lucas lose a lot of his money
and make him look bad to the entire world
and you could be well known and liked

None of us here like Lucas or Lucasfilm.
I have death wishes on Lucas and Macullum.
we could all probably get 10s of thousands of dollars!
Author
Time
Post link
Frame counts would be completely different as NTSC is 29.97 fps and PAL is 24 fps.
Author
Time
Post link
Yes and no. Theoretically, frame counts should be the same across both formats regardless of fps, as the '04 discs should be encoded progressively. However, the OUT discs may well be interlaced (at least for NTSC) so the video frame counts would quickly mismatch. But either way, you'll need separate patches, due to resolution and audio differences.
Author
Time
Post link
Just wanted to mention that I've made a little progress on the decoder. Currently it can decode the program stream and pack header/pack. It is able to determine SCR and the type of packet (audio/subpicture, video, padding, etc). It is also able to work with a group of VOB files and move to the next when needed. This is good because then we don't have to worry about file size limits.

So, not very much progress, but it's a start. The next thing I need to do is calculate PTS and DTS and figure out the timing of things. I did an initial test where if the SCR moved backwards, it threw away packets until time moved forward again (to see if that would fix the scroll), and it sorta did, but also other big chunks of the movie were missing, so I have more to learn about timing. Also, there were picture artifacts at the transitions, so that worries me. I was hoping that packets were complete pieces of the movie (I-frame to I-frame for example), but it appears that isn't the case. Anyhow, it was just a quick test to see what would happen... we wouldn't want such a brute force method anyways, because it eliminates choices. If the user wants to keep streams etc, that should be no big deal.

The decoder is written in python: the language is readable, portable (Linux, Windows, cygwin, OSX, etc, we are covered), and doesn't have to be compiled. The program is mostly doing I/O operations, so the tiny bit of interpreted processing happening isn't slowing things down. It takes 2 1/2 minutes to run through each 1GB VOB file on my system (at the same time writing that modified VOB).
Author
Time
Post link
Originally posted by: ThatArtGuy
Frame counts would be completely different as NTSC is 29.97 fps and PAL is 24 fps.


I'm sure you meant PAL is 25 fps...
Author
Time
Post link
Uh... yeah... I *did*. Thanks!

(Yeah, yeah, I know. I messed up. First mistake I ever made.)
Author
Time
Post link
I can think of another unresolved detail - though you may have thought of it - the video may not line up between the theatrical and 04 discs. For instance the exact position of the movie on screen may not stretch all the way to both sides... in "patching" you may want to resize the video from one of the DVD's to match the other (it would also be good to give people a choice about which way to do this, for instance splicing the 04 disc into non-anamorphic resized to match the theatrical discs - or splicing theatrical disc into anamorphic resized to match the 04 disc). I only suggest this because if people are wanting to apply relatively "few" patches to the theatrical disc they may not want to have the rest of the theatrical disc re-encoded ... but those using the '04 disc as a base to be patched probably would want to resize the theatrical disc instead.
Some were not blessed with brains.
<blockquote>Originally posted by: BadAssKeith

You are passing up on a great opportunity to makes lots of money,
make Lucas lose a lot of his money
and make him look bad to the entire world
and you could be well known and liked

None of us here like Lucas or Lucasfilm.
I have death wishes on Lucas and Macullum.
we could all probably get 10s of thousands of dollars!
Author
Time
Post link
Oh and adding "burnt in" subtitles would be something I'd like to offer as a suggestion for a patch.
Some were not blessed with brains.
<blockquote>Originally posted by: BadAssKeith

You are passing up on a great opportunity to makes lots of money,
make Lucas lose a lot of his money
and make him look bad to the entire world
and you could be well known and liked

None of us here like Lucas or Lucasfilm.
I have death wishes on Lucas and Macullum.
we could all probably get 10s of thousands of dollars!
Author
Time
Post link
Originally posted by: boris
I can think of another unresolved detail - though you may have thought of it - the video may not line up between the theatrical and 04 discs. For instance the exact position of the movie on screen may not stretch all the way to both sides... in "patching" you may want to resize the video from one of the DVD's to match the other (it would also be good to give people a choice about which way to do this, for instance splicing the 04 disc into non-anamorphic resized to match the theatrical discs - or splicing theatrical disc into anamorphic resized to match the 04 disc). I only suggest this because if people are wanting to apply relatively "few" patches to the theatrical disc they may not want to have the rest of the theatrical disc re-encoded ... but those using the '04 disc as a base to be patched probably would want to resize the theatrical disc instead.

I hadn't considered using the OUT disc as a base, but it won't be hardcoded, so it shouldn't be a big deal: I will have a configuration file for each movie (so 3 of them, using 2004 as a base). You could make 3 more, using OUT as a base. This will make it possible for others to use the tool in future projects unrelated to Star Wars. It would just be a matter of configuration and project set-up.

Originally posted by: boris
Oh and adding "burnt in" subtitles would be something I'd like to offer as a suggestion for a patch.


Of course. This should be covered by the partial frame re-encoding feature I hope to include. I did some reading on the subject, and this technique is mentioned on the doom9 boards, with a hardware device given as an example that it can be done (it put studio logos/overlays on digital footage, etc). It is very encouraging to me that someone has been able to pull this off. I wasn't sure it was even possible.