logo Sign In

Info: Gigapixel AI vs infognition Super Resolution / What to use to upscale SD to HD or 4K

Author
Time
 (Edited)

Hello everyone!

Upscaling SD content to HD or even 4K is something that crops up from time to time. I myself have at least some experience with up-scalers, with my preferred method involving using infognitions Super Resolution plugin for VirtualDub for live action stuff and SuperResXBR for animation. However, I recently became aware of a new up-scaler called Gigapixel AI. https://topazlabs.com/gigapixel-ai/

Needless to say, I grew quite curious about this program. Now, nowadays you can get the Super Resolution plugin free of charge, but to get Gigapixel AI, you need to pay at least $84.99 so one wonders if it’s worth it.

That’s why I decided to have a discussion about it here.

As for how to use Gigapixel AI for video, there’s an excellent tutorial here: https://captrobau.blogspot.com/2019/05/tutorial-upscaling-video-with-topaz-ai.html
The same guy who wrote that tutorial has experimented with the program to upscale Star Trek: Deep Space Nine to HD and even 4K:

https://captrobau.blogspot.com/2019/03/remastering-star-trek-deep-space-nine.html

Author
Time
 (Edited)

I’m seriously curious to see how this could improve cartoons.

Author
Time

Jesta’ said:

I’m seriously curious to see how this could improve cartoons.

Well, the guy I mentioned in the first post has used Gigapixel AI to make an HD mod of Final Fantasy VII. Click on the image for a trailer. Remako Trailer

Author
Time

Now, this is interesting:
https://www.blu-ray.com/news/?id=25179

newly remastered HD footage from the television series

That suggests that Deep Space Nine either is or was remastered in HD, but for whatever reason (probably the poor sales of TNG Blu-rays), it’s never been officially released before now.

Author
Time

Sorry for the inactivity, but I think now is a good time to start a fundraiser so that I can buy Gigapixel AI and compare it against infognitions Super Resolution. Just donating $1 will help. PM me for details.

Author
Time

I’ve been trying out gigapixel ai on my projects, and it works tremendously well. I previously used Red Giant’s Instant 4K and Waifu2xPhoto. There’s also ESRGAN, but I don’t currently have a machine that can run it.

You can download it for a free trial and it has no limitations.

Author
Time

I’ve been recommended by a smart dude something that may be even better: ESRGAN/BasicSR. It’s an AI-upscaling algorithm just like Gigapixel, but you can train it with your own datasets, so if you have a representative dataset of what you’re trying to upscale (you need high res and low res reference for training), you can get really incredible results.

I’m currently playing around with it, and while training is very slow (we’re talking many days), it’s impressive what it can do imo.

Author
Time

TomArrow said:

I’ve been recommended by a smart dude something that may be even better: ESRGAN/BasicSR. It’s an AI-upscaling algorithm just like Gigapixel, but you can train it with your own datasets, so if you have a representative dataset of what you’re trying to upscale (you need high res and low res reference for training), you can get really incredible results.

I’m currently playing around with it, and while training is very slow (we’re talking many days), it’s impressive what it can do imo.

I’ve tried some pre-trained ESRGAN upscalers, and I’d say it is a more natural looking upscale than ai gigapixel, but I’d still say ai gigapixel wins in the detail, even if it looks a bit too sharp. And I’d still say in terms of noise reduction ai gigapixel and waifu2x are better at removing compression artifacts. Of course waifu2x doesn’t try and add or recover detail, but simply a high resolution upscale that retains all the original detail.

All three are really good tbh, and all have their purposes. I’d say a mix of them is ideal.

Author
Time
 (Edited)

emanswfan said:

TomArrow said:

I’ve been recommended by a smart dude something that may be even better: ESRGAN/BasicSR. It’s an AI-upscaling algorithm just like Gigapixel, but you can train it with your own datasets, so if you have a representative dataset of what you’re trying to upscale (you need high res and low res reference for training), you can get really incredible results.

I’m currently playing around with it, and while training is very slow (we’re talking many days), it’s impressive what it can do imo.

I’ve tried some pre-trained ESRGAN upscalers, and I’d say it is a more natural looking upscale than ai gigapixel, but I’d still say ai gigapixel wins in the detail, even if it looks a bit too sharp. And I’d still say in terms of noise reduction ai gigapixel and waifu2x are better at removing compression artifacts. Of course waifu2x doesn’t try and add or recover detail, but simply a high resolution upscale that retains all the original detail.

All three are really good tbh, and all have their purposes. I’d say a mix of them is ideal.

See, that’s where your own training data comes into play. Gigapixel is trained on a generic dataset you know nothing about. Of course they have probably put lots of effort into a good curated dataset for a good product, more work arguably than people working for free have done for ESRGAN. Gigapixel may be good at noise reduction, but it’s not perfect, because it may not apply perfectly to whatever dataset you yourself have. To give an example, I’m currently training ESRGAN to upscale old roq game videos (Quake 3 engine). For that I took a video I have in high quality (Prores) and made roq encodes at similar bitrates to the game videos. Then I put that into the training. I’m not far into the training yet (only 2 days or so), but it seems promising. I tried a screenshot from one of those videos in Gigapixel and while it was sharp, it misinterpreted a lot of the artifacts as detail, and I think that with enough training, ESRGAN will be much smarter about upscaling this particular kind of source.

Author
Time

I recently discovered that the developers of Gigapixel AI have begun testing a video upscaling app. For the moment, it’s only for Windows 10, but during the beta test, you can try it out for a non-commercial purpose free of charge. For the curious, here’s a link: https://videoai.topazlabs.com/

Author
Time

After some testing of the video app, I can safely conclude that I’m sticking with infognitions Super Resolution plugin whenever I want to upscale live-action footage for now on. Why is that, I hear you ask? Well, I decided to upscale the deleted scenes from Star Wars: Episode I and what I’ve got, in the end, was this: https://drive.google.com/open?id=1QxRpCOvvqQ3WGi0klL-yhfV_-d-oqpwe
See, I noticed that while Gigapixel does an outstanding job on close-ups, it falls flat on its face when it comes to faraway objects. As you can see, the faraway objects, at least in my opinion, look like cartoon blobs.

Author
Time
 (Edited)

It seems to have amplified video artifacts (edge unsharp) and not real detail… Limitations of the source material I think.

Author
Time

Topaz Gigapixel has just released a new app, Video Enhance AI.
I am already using the Windows Beta Gigapixel AI for Video.
I will be trying the new app soon.

Please check it out and let us know your thoughts.

Thanks

Author
Time

phoenixobia said:

Topaz Gigapixel has just released a new app, Video Enhance AI.
I am already using the Windows Beta Gigapixel AI for Video.
I will be trying the new app soon.

Please check it out and let us know your thoughts.

Thanks

$199 or $299 after the sale ends? I’ll consider it if someone can help me out with the finances. And I assume it’s a one month trial.

Author
Time

I know, nobody wants to hear, but…
Just to assure that I am no newbie in all this, I do film restoring (editing, color correcting, encoding a.s.o) professionally since almost 20 years.
As all of you know, there is no real way to upscale anything really (except for vector graphics…), because you cannot generate any detail that isn’t there, out of nothing. There are many cases where there ARE more details, but not obviously visible, where you can do a nice job with the right sharpen algorithms, but also this is no “real” upscaling. Clear.

This was the situation, until somebody had the idea to use AI…
Due to rather quick inventions and later “central” implementations, a lot of developers presented a lot of software, that at the end of the day mostly used the same algorithms, and this looked very, very intersting in the beginning. There were and are samples over samples that look unbelievably good, quite magic.
So I tested everything I could get - whenever I found the time for it - I thought: First see, then think. If it worked really well, the algorithms are not worth to think about, at least for me, who did not intend to take part in this by coding, just as a user.

The first upscale I did (after a lot of testing I decided to use Gigapixel for it) was for a reproduction of an old painting, to get a more detailled printing master. This worked incredibly well! In THIS case…
It worked, because the brushstrokes are some kind of pattern that made it easy for Gigapixel to find SIMILAR PATTERNS in its database. And this is what it’s all about in the end, and with all machine learning:

To find a similar pattern that you can use for details! But…

This finding similarities is done by accident! Only similarity counts! That means, that if the structure of a distant grass valley for some reasons looks like nearby green coloured hair, then the machine learning algorithm MAY decide to use this structure and add details to the grass from somebody’s green hair.
You MAY not notice this in a picture - sometimes you don’t, somtimes you do, accidentally.
But in a video you always notice it, because the used patterns CHANGE all the time. That means, in one frame AI uses different patterns than in the next ones. So this generates a lot of noise, unsteadiness, flicker a.s.o.
I made a lot of tests with it, wrote some not too bad avisynth-scripts to compensate these effects - not useless, but also not convincing in the end. No real upscaling possible, least of all upscaling old, not so good material, which would have been a great goal!

The newer thing about real video-AI-upscaling then was that they tried to use
-frames from environmental frames of the same scene as part of the database, a wise decision to find really similar structures, but at the cost of not finding more sharpness in most cases…
-implement more steadiness in the selected patterns
So I tested f. e. Gigapixel’s video-tool that one mentioned above.
As expected the winning of details was MUCH less than with image-upscaling. Comparable to a rather medium conventional-“Upscale” quality.

So I began to think about the situation. My opinion: The problem with all this is:
Added details are still added by simple SIMILARITY, regardless of what origin they have!
So this all is still no real AI, that would work similar to a human restorer. When a restorer adds a lost detail by f. e. restoring a painting, he knows WHAT he restores! F. e. some grass in the distance…
So, what is needed in the future is, to build real HUGE nested databases of all the things you can see out there,
with all nesting, linking, rating, comparing, a. s. o. that a human brain does. So that an algorithm in the end will KNOW that it has to take the RIGHT pattern, and not something that looks just similar.
Until then we will sure ecperience some improvements - but if you really look at close range, it will always have some nature of Frankenstein.

Author
Time
 (Edited)

FrankB said:

I know, nobody wants to hear, but…

You are right. Those details don’t exist but I don’t think anyone expects it to be as good as a human restorer.
I tested the new app and I am amazed by the results. It’s even better than the Beta app.

I’m also not 100% sure if Gigapixel checks for similar patterns or objects already registered, but by what’s available on the picture itself and sort of digital painting new pixels. but I don’t know. I’m just observing the process and comparing.

I will post a side by side picture of the upscaling results for my Beauty and the Beast LD for comparison. I was so skeptical, but what I see is more than I hoped for. To me it’s still better than some of the terrible restoration done by the humans at Disney mostly because of certain decisions they made to alter the original. 😃

Author
Time

I agree to your last sentence. 😉
How exactly did you upscale you BatB-LD?

Author
Time
 (Edited)

FrankB said:

I agree to your last sentence. 😉
How exactly did you upscale you BatB-LD?

I use a Panasonic LX-101 Laserdisc player and a Blackmagic Intensity Shuttle USB 3 device to capture from the composite source into either an uncompressed or a high bitrate compressed SD format. I don’t use the device’s upscaling feature as it appears to just output a bicubic resized image. There are a couple of settings for Chroma and brightness at capture level that I have slightly adjusted due to a very subtle Brightness/chroma burn I notice. The captured video is an interlaced 29.97 fps video. In this case it’s telecined.

Then I use TMPGEnc Video Mastering Works to deinterlace (IVTC) and crop the letterboxed picture. I convert it to another lossless or high bitrate SD format using the highest processing options available to use for upscaling later, as Gigapixel currently can’t handle interlaced footage well. The converted video is a progressive 23.976fps video. The TMPGenc app has impressive advanced settings for video encoding, deinterlacing, and good filters which I don’t use at this stage.

I then load the progressive SD video in Gigapixel Video Enhance AI (using a 30 day trial for now). There are 3 upsampling methods to choose from. In this case I used Upsampling(HQ-CG) which I think looked better for BatB, plus HD size and the output format.

Currently there is mpeg4, 8bit tif, 16bit tiff, and png outputs in Video Enhance AI but I see no setting for mpeg4 to change the bitrate or CBR/VBR (could be the limitation of the free trial, not sure). Final result is high quality but the file size is not in your control.

You can consider this your final video but since I want more control and I also have enough storage, I choose png as my output to keep it lossless for further processing. This method outputs each progressive frame as separate png files in sequential filenames saved in a folder (300+ GB).

Then I load the first image/frame in TMPGEnc Video Mastering Works and it automatically loads the other frames to form a video. If any filters needed such as denoise, sharpness, etc, I find it’s better used at this stage on the already upscaled picture. I choose MP4 with H.264 codec.

In the output settings I make sure to choose 23.976 fps progressive otherwise it will change the speed of the video when dealing with still images as frames.
I choose VBR for 2pass encoding so I have control over the file size.

When the video is ready, I use MKVToolNix to add the audio to the video from the progressive SD video I created earlier. This avoids further processing/encoding of the video and only adds the audio to it in a couple of minutes.

That is my final version of the video. 😃

I’m still testing different things but so far this has given me great results.

Sorry for the long reply 😉

Author
Time
 (Edited)

Sounds as if you tried a lot not to lose anything around upscaling. Really good is, that you exported from GPVE to lossless images, to collect these together afterwards. How well does the heart of this, GP Video Enhance, seem to work for you? I found it rather weak. Can you post some screens?

May I admit some things? I feel free to do so, maybe I can help:

For IVTC, one of the key points in your workflow, there are much better ways. Most avisynth-experts use TIVTC which produces excellent results without any loss in quality if you use avisynth correctly. Even better is to IVTC by hand with avisynth (even if the doom-cracks don’t like this… 😉 ) Very often you have only one, or less than 4 or 5 pattern-changings to remove, which is by hand done quite fast, and you get 100% jitter-free results, which NO automatic algorithm can achieve.
Also, unfortunately, if telecining (pulldown) had been done directly while scanning and later different compressions were made, there often remain “staircase”-artefacts (don’t know the right term in English) after IVTC. Hard to correct these without losing resolution, but if you plan to upscale/sharpen afterwards, often better to remove it at the cost of a bit less resolution, which you “get back” (not really) with your upscale. Otherwise these staircases will be more and more visible. If you have a source with absolutely NO such artefacts after IVTC, you are lucky.

At the end I would rather export to some lossless codec, then it is possible to later improve something, edit something a. s.o. But maybe you just didn’t mention it.

Author
Time
 (Edited)

FrankB said:

Sounds as if you tried a lot not to lose anything around upscaling. Really good is, that you exported from GPVE to lossless images, to collect these together afterwards. How well does the heart of this, GP Video Enhance, seem to work for you? I found it rather weak. Can you post some screens?

May I admit some things? I feel free to do so, maybe I can help:

So far the results are satisfactory to me as I have been comparing different upscaling methods. GPVE gave me more noticeable details and sharpness. I have comparison images ready if I can post here.

Of course, help is always appreciated as there are different ways of doing this and I’m constantly looking for ways to learn and improve, and here’s the place to share our experience and knowledge. 😃

As for IVTC, I haven’t tried Avisynth yet but I’ve read about it and would try it. I am aware of the change of pattern that can happen but I think that could be the case with any algorithm. As you mentioned, doing those by hand to produce progressive results is the best way to have perfect results but I don’t see how it can be fast. Shouldn’t that take forever? Please elaborate on that.

Second, I’m not sure I understand the part you talk about telecining while scanning.
If the original footage has been 24fps film/animation and telecined (3:2 pulldown) to 29.97fps which is the case with Laserdiscs, IVTC is the process to get rid of those added frames and turn it to 23.976fps. The results should have no jagged edges or as you say staircase-artifacts and no half resolution. My IVTC video has no rough edges if that’s what you mean, so please explain this.

Lastly, Having a lossless video at all times is good. I only do the last conversion after I’ve done all the edits and improvements and when there are no further edits needed, I encode it to my final video because Keeping the lossless files takes a lot of space.

Below are the images. It’s best to download and see at 100% but you can still see the difference here.
Beauty and the Beast LD upscale comparison 1
Beauty and the Beast LD upscale comparison 2
Beauty and the Beast LD upscale comparison 3
Beauty and the Beast LD upscale comparison 4
Beauty and the Beast LD upscale comparison 5
Beauty and the Beast LD upscale comparison 6
Beauty and the Beast LD upscale comparison 7
Beauty and the Beast LD upscale comparison 8
Beauty and the Beast LD upscale comparison 9