Sign In

Star Wars 4K77 - Regraded - No DNR (Released) — Page 2

Author
Time

FYI
I’ve been downloading this since the day it was posted here. I’ll keep sharing it till there’s no one downloading. But it’s basically stalled right now, not downloading. I see two people active: one with 44% and the other with 0%

Author
Time

Well, I am seeding. I’m gonna seed today overnight. My internet connection is not exactly faster than the light or anything, but I’m not going anywhere. I might turn the PC off tomorrow night, and some others nights, but I’ll be back in the morning.

Author
Time

So is just the 1080 going up?

Wouldn’t waiting for the “original” files being regraded first be a better way? That way it’s not just based on the 1080 mkv? Either way, great that this is happening and thank you all for the work on all this!

Author
Time

Colors look great on this. Well done.

Thanks for sharing the project files too.

Author
Time

ChainsawAsh said:

If you’re going to be doing any reencoding, particularly for color alterations, it’s always going to be a better idea to convert it to a lossless (or near-lossless) codec first. H264 is… not that.

I thought CRF=0 was lossless for h264/x264. That’s what I’ve read everywhere on the web at least.

Author
Time
 (Edited)

stretch009 said:

ChainsawAsh said:

If you’re going to be doing any reencoding, particularly for color alterations, it’s always going to be a better idea to convert it to a lossless (or near-lossless) codec first. H264 is… not that.

I thought CRF=0 was lossless for h264/x264. That’s what I’ve read everywhere on the web at least.

Technically, I believe CRF 0 is only “visually lossless”. Meaning with the naked eye you shouldn’t be able to see any difference, which is not the same as a true lossless codec like Lagarith, huffy or MagicYUV. A truly lossless codec is like using winzip or winrar - just as you can unzip or unrar to restore the original uncompressed file, you can take an uncompressed AVI, compress it with one of these codecs and then take that output file and resave it as an uncompressed AVI and the MD5/HASH will be identical to the file you started with. You won’t get that with h264 CRF 0.

http://www.thestarwarstrilogy.com

http://www.the007dossier.com

Author
Time

I’m a little bit confused…
According to several reference sites, CRF=0 is true lossless (in the same color space).

https://trac.ffmpeg.org/wiki/Encode/H.264 said:

The range of the CRF scale is 0–51, where 0 is lossless

https://www.oodlestechnologies.com/blogs/H.264-Basics-%3A-CRF-Guide said:

0 is lossless (Lower the value, higher is the quality)

https://forum.videohelp.com/threads/358901-is-x264lossless-really-lossless said:

Yes it’s truly lossless in the same colorspace
(…)
When people talk about “lossless” codecs in terms of video, it refers to the decoded output. So the decoded output of say, a lagarith lossless encode in YUV 4:2:0 will be the same as the decoded output of x264 crf0 YUV 4:2:0 using intra only, or the decoded output of x264 crf0 YUV 4:2:0 long GOP + some slower settings. The only difference in terms of quality will be the last one will be smaller in filesize . The decoded output and uncompressed filesize will be exactly the same for all 3

Author
Time

H.264 is lossy. True lossless video is crazy huge. If we had true lossless video it would take many orders of magnitude more storage.

Williarob has the real files. They are too large to share anywhere unless you want stacks of hard drives. Literally. Big ones. We’re talking about loading this kind of stuff into decently large SANs.

Each frame has been said to be about 100MB. I don’t know the frame count, but let’s say 150,000. That’s 15TB. That’s not going to fit on a Blu Ray.

H.264 is to video what mp3 is to audio. Or jpeg vs raw. They use many tricks to compress the original source.

Point is, he’s got the lossless scan of film frames. It’s a totally different ballgame to work with that stuff. Not something most people work with outside of serious pro video and movie setups. This stuff is seriously pro level.

Author
Time
 (Edited)

Uncompressed Star Wars was 21 TB. However, it could be losslessly compressed to about a third of that size.

Regarding whether or not h264 is truly lossless, I did a quick test. I took a short, 3 second HD video and saved it as an uncompressed AVI. Then I took the uncompressed AVI and saved it with lagarith lossless compression as a new AVI file. I opened the new compressed AVI file again, and saved it as a third uncompressed AVI file. Using Hasher, I created checksums for both files. Both The original uncompressed file and the uncompressed lagarith file are identical:

Imgur

Then, following the directions on the FFMPEG website for lossless encoding, I did the same thing. Here is the command I used to create the “lossless” file:

ffmpeg -i “q:\bombardier-captain-vincent-eva-Original.avi” -c:v libx264 -preset ultrafast -crf 0 “q:\bombardier-captain-vincent-eva-ffmpeg.mkv”

Then I took that MKV and resaved it as an uncompressed AVI. Comparing the hash values, it is clear that this is not the same file we started with:

Imgur

I tried it again using Handbrake, and the checksums still didn’t match.

QED h264 is only visually lossless. Unless you can uncompress it and get the exact same file you started with, it is not truly lossless - just like with winzip or winrar - you should be able to get back the exact same file you put into it. If you can’t, then you lost something.

EDIT: This is probably due to a behind the scenes color space conversion: Uncompressed RGB -> YUV420; But if it can only losslessly compress data in the same color space that’s pretty limiting.

My preferred lossless codec is MagicYUV (https://www.magicyuv.com/) because it is very fast and can losslessly save even 10-bit+ color formats.

http://www.thestarwarstrilogy.com

http://www.the007dossier.com

Author
Time

There we have it. 21TB. That’s awesome.

I think it’s more than just color space. Most video algorithms are using a huge amount of tricks to get file sizes down. Hence why we have hardware acceleration to decode in many cases. It’s fairly complicated to do, and still has to send a massive amount of data on actual output to appear to be of good quality.

(In basic terms) video encoding is using differences between frames, not only reusing color bit info, but everything else. Distances between pixels between frames, and so forth. The very early mpeg4 stuff was really interesting and they’ve improved a lot over the years. Even mpeg2 is doing more than most people realize. Neither is lossless. It always surprises me that people don’t grasp this. That OTA TV, for example, isn’t lossless. It’s just that it hasn’t been re-encoded by your provider. Same with Blu Ray, even at 4k. It’s still lossy, it’s just at the highest they can get for the length of content. Similar to a 320kbps mp3. They do, however, have room for awesome lossless audio on Blu Ray these days, which is very nice.

Thank you for pointing out what true lossless compression is about. I’ve dealt with it in the audio world more over the years but also understand video decently. As with rar, zip, flac audio is another easily understood example.

Author
Time
 (Edited)

Incidentally: that 3 second uncompressed HD avi I was playing with was 450 mb. Compressed with h264 (CRF 17) it was 1.5 MB.

Lossless h264 version was 8 mb and the lagarith version was 30 mb.

Interesting experiment.

However we should probably get back to talking about what a great job Sanjuro did with his color correction… There is probably a thread in another section of the forum for discussing compression.

http://www.thestarwarstrilogy.com

http://www.the007dossier.com

Author
Time

Agreed.

Any chance this correction will be done on the original source? I’m sure the 1080 looks great, but I’d be interested in seeing it applied to the source material and rendered out from that. I guess for the sake of it being more direct than re rendered from the “lossy” format.

Either way, will have to track this down. Haven’t seen it hit spleen or newsgroups yet.

Author
Time

Is there going to be a DNR version of this uploaded?

Author
Time

Likely not, as the DNR version had additional cleanup work and a different color correction from the non-DNR version. I’m sure you could take the DNR version and get the colors close to this (maybe Dre’s color matching tool would work?), but you’d have to start over and generate new LUTs - the ones shared here for the non-DNR version would give you different output colors since the starting colors aren’t the same.

a trolling bantha

Author
Time

Williarob said:

I already rendered out the project in 4K and 1080p from the original source. I don’t think it’s made it out into the wild yet but you can find it on the forums at thestarwarstrilogy.com

That’s great news! Very much look forward to getting a chance to see it!

Thought the originals were pretty darn good, but could see a few places that might’ve had room for some different choices.

Thanks for the reply and all the hard work on all of this. Still need to donate a bit more to the '83 when I get a good opportunity to do so.

Author
Time
 (Edited)

This is some really nice work, and in record time! Congrats! I think your shot by shot can also be used as a great basis for further tuning to cater to each and everyone’s personal preferences. Here’s my little effort, starting from the last shot of the film. I personally feel the color grading could use a little warmth to bring it more in line with the technicolor references.

sanjuro_61:

DrDre:

I applied this adjustment to the two sets of sample frames I found:

It might need a little tuning here and there, to get to a final version (I see a few frames with some excess magenta), if there’s an interest, but you get the general idea.

Here’s the LUT:

https://drive.google.com/file/d/1hDo_XIiozzDanx5B1WAn-GuG31UwHlYW/view?usp=sharing

Author
Time
 (Edited)

Just a few notes about the (hypothetical) sharing of lossless masters, if it’s of interest to anybody:

  • crf 0 is the wrong way to do lossless x264 afaik, the correct way is -qp 0 (as crf is conceived to achieve variable bitrate with varying qp (quantization parameter) across blocks/frames. QP explicitly sets it to 0 globally.

  • x264 only encodes YUV values. A conversion from RGB will automatically lead to rounding loss and thus not be 100% lossless anymore (but still good enough practically). ProRes also works in YUV.

  • x264 by default encodes in 4:2:0. Compare to RGB, which naturally is 4:4:4. So you automatically lose half the color information. The solution is to encode in the 4:4:4 mode of x264, which has to be explicitly stated.

  • Normal x264 only has 8 bits, whereas ProRes has 10 bits (or even more with a proper Apple ProRes encoder). For sharing a master for proper color correction work, a higher bit depth would be highly beneficial. This can be achieved with the 10-bit version of x264.

  • Lagarith as well as normal lossless AVIs only support 8 bits as well, therefore are not useful for sharing lossless masters, especially not if further color work is to be done.

  • Lagarith has 3 modes afaik: RGB, RGBA and YV12. YV12 is a 4:2:0 mode and tends to produce significantly smaller files than RGB. RGBA includes alpha channel, should be irrelevant for most cases. RGB is default. The mode can be changed in the codec settings (for example in VirtualDub when selecting Lagarith).

  • x264 is a DCT-based codec, so everything always gets converted into the frequency domain on a per-block basis, which of course can introduce rounding errors in theory, especially at 8 bit. It’s possible they have ways to avoid this, but it’s something to keep in mind. Same applies to ProRes by the way, also DCT-based; but ProRes is not meant to be lossless to begin with, so that’s okay.

  • ProRes, unless you use the 4:4:4 mode, also is 4:2:2 and thus not ideal for anything that will require further color work imo.

  • Currently the only way (known to me) to have a truly lossless master with high bit depth is an RGB 16-bit TIFF/PNG/(some other formats) sequence or some exotic AVI codecs that support higher bit depths, but aren’t widely supported or maintained.

Of course it’s still possible that x264 it’s not 100% lossless, because it was never really made for it and who knows what it does to the signal behind the scenes, but some people have evidently achieved losslessness in x264, see here: https://stackoverflow.com/questions/6701805/h264-lossless-coding

Either way, it’s certainly good enough, if you use the 10-bit version, in addition to qp 0 and 4:4:4 color setting. But at that point the bitrates will quite likely even exceed those of ProRes (as I believe ProRes in High profile has the equivalent of qp=4 if I’m not mistaken, at least that’s what ffmpeg shows), and come at the cost of being several times more inefficient than ProRes when it comes to editing. Most editing software doesn’t even support importing x264 10 bit, so you would have to convert it back to ProRes or the likes anyway in order to edit it.

With that said, I think that if you drop the strict “lossless” requirement, we could share very high quality masters (comparable to ProRes) suitable for color grading & VFX if we just set the qp to values above 0. Again, with 10 bit x264 in 4:4:4 mode. As a starting value, I would probably try qp 4, but maybe it’s possible to go even higher. To test the quality of the result, you’ll need to do 16 bit screenshots (as PNG/TIFF don’t support 10bit), for example using AviSynth+ & VirtualDub, and then I typically load that stuff into Photoshop and play with the Levels to blow up dark areas to see if any artifacts become noticeable.

I think this way there could be a notable size improvement over ProRes, which would help with transferring. The recipient could convert back to ProRes then for his own work.

Author
Time

TomArrow said:
x264 is a DCT-based codec, so everything always gets converted into the frequency domain on a per-block basis, which of course can introduce rounding errors in theory, especially at 8 bit. It’s possible they have ways to avoid this, but it’s something to keep in mind.

Yes. This is why I’m not convinced that x264 is truly lossless.

Author
Time

Chewtobacca said:

TomArrow said:
x264 is a DCT-based codec, so everything always gets converted into the frequency domain on a per-block basis, which of course can introduce rounding errors in theory, especially at 8 bit. It’s possible they have ways to avoid this, but it’s something to keep in mind.

Yes. This is why I’m not convinced that x264 is truly lossless.

Yeah, me neither. But it’s certainly good enough for almost any scenario imaginable, except maybe some scientific uses where it really needs to be bit-perfect.

Author
Time

DrDre said:

This is some really nice work, and in record time! Congrats! I think your shot by shot can also be used as a great basis for further tuning to cater to each and everyone’s personal preferences. Here’s my little effort, starting from the last shot of the film. I personally feel the color grading could use a little warmth to bring it more in line with the technicolor references.

sanjuro_61:

DrDre:

I applied this adjustment to the two sets of sample frames I found:

It might need a little tuning here and there, to get to a final version (I see a few frames with some excess magenta), if there’s an interest, but you get the general idea.

Here’s the LUT:

https://drive.google.com/file/d/1hDo_XIiozzDanx5B1WAn-GuG31UwHlYW/view?usp=sharing

Cool - thanks Dre!

http://www.thestarwarstrilogy.com

http://www.the007dossier.com

Author
Time
 (Edited)

DrDre said:

This is some really nice work, and in record time! Congrats! I think your shot by shot can also be used as a great basis for further tuning to cater to each and everyone’s personal preferences. Here’s my little effort, starting from the last shot of the film. I personally feel the color grading could use a little warmth to bring it more in line with the technicolor references.

sanjuro_61:

DrDre:

Hey DrDre, I was just wondering, are the technicolor references really that warm? Just looking at the video scopes, the red is very pushed. Not saying that’s wrong, I defer to your knowledge, just curious! The sanjuro_61 frame is a bit cool according to the scopes, so just as a quick test, I balanced the highlights, and the result is somewhere in the middle.

Author
Time

poppasketti said:

DrDre said:

This is some really nice work, and in record time! Congrats! I think your shot by shot can also be used as a great basis for further tuning to cater to each and everyone’s personal preferences. Here’s my little effort, starting from the last shot of the film. I personally feel the color grading could use a little warmth to bring it more in line with the technicolor references.

sanjuro_61:

DrDre:

Hey DrDre, I was just wondering, are the technicolor references really that warm? Just looking at the video scopes, the red is very pushed. Not saying that’s wrong, I defer to your knowledge, just curious! The sanjuro_61 frame is a bit cool according to the scopes, so just as a quick test, I balanced the highlights, and the result is somewhere in the middle.

Yes, the technicolor prints are very warmly graded, although there’s a little too much magenta/red in the highlights. I will provide an update later today, to correct this.