I have been kicking around an idea for a sort of end-all, be-all technique for recruiting image detail from film grain. I briefly described it on the 35mm thread and talked with Dr Dre about it as well. The somewhat involved explanation in my original message is below, but first, some background on the film process and film grain:
So let’s take Mike’s Legacy project as an example of how detail recruitment is done in its current best-case scenario. From my message to Dr Dre:
Mike’s method involves stacking up to five of the same frame on top of each other from sources of varying quality, and doing a weighted average of the pixel values. He also recruits data from neighboring frames, but let’s just focus on the stacked frames for now. So say he has 5 stacked frames. Two are slightly sharper (Tech frames) and 3 are slightly softer (Kodak frames). If you do a weighted average of every pixel, the softer frames will tend to override the sharper frames, since there are more of them. The result is a cleaner, but softer image. If you’re looking to retain detail, your best bet is to stick with the sharpest frame and discard the rest, since any averaging will invariably soften the image, regardless of the increase in clarity.
I think there’s a way to keep both the detail and the grain-free look. You would probably need a scan that is in 8-10k quality, so that each grain (more accurately each dye cloud) is distinguishable from another, at least mathematically. It could be that a 4k scan may have this level of detail. In any case, it should be possible for a sufficiently robust algorithm to examine the frame and identify the center of each dye cloud, recognizing local minima in luminosity for each color layer. With this map generated, it makes transparent the pixels not directly surrounding the center of each dye cloud, so you have in effect made cheesecloth of the image. You keep only the center of the dye cloud, information that is the most likely to have come from an actual photon impacting the silver halide crystal at the center of the dye cloud. In a way, the map should contain all of the actual color and luminosity information necessary to digitally ‘develop’ a new image by expanding each dye cloud back to its original size. However, with multiple stacked images, this process is repeated for each one and the results of this are overlaid, with the sharpest image on top and the softest image on the bottom.
This process could also be applied to neighboring frames in a more traditional super resolution method. If the center of two dye clouds is targeted to the same pixel, then and only then should a weighted average of the pixels be applied.
TL;DR version: Taking the pixels from a high resolution scan that are most likely to contain actual image information and discarding the space in between dye cloud centers, then overlaying multiple prints or sequential static frames, you should theoretically be left with an image with much higher detail and sharpness, since only the noise is removed, and the detail is multiplied over the number of sources used.
It would be interesting to run an experiment to test this theory. One would need to take a sequence of photos of the same object, preferably with grainy film stock, then develop them and scan them with a high resolution film scanner. After that, you would need a program that identified local luminance minima (the darkest parts of the grain are where a dye cloud formed) then applied a weighted transparency to any pixels not in these sections. Process the images, then overlay them, and compare the result with the same image stack run through a conventional super-resolution algorithm. With enough frames, one could upscale the resolution with a similar jump in actual image detail until running into the limit of resolution dictated by the quality of the camera lens.
This process, if it works, could perhaps be used on the infamous speeder and sandcrawler shots to bring them in line with the rest of the film.