Sergei Mikhailovich Prokudin-Gorskii was one of the first to capture the world in color; with his beam splitting camera, he captured the intensity of different color channels (R, G, B); printed them on red, green, and blue film, and projected them on top of each other to create colorful images!
To optimally align plates with smaller pixel values, we can simply slide each R, G, and B channel along each other until we find a good “fit” — much like you would do if you physically had the film and wanted to align them on a projector.
The metric for a good fit comes down to aesthetic choice; we explored a few metrics:
We ended up mostly using the SSIM Metric; rather than doing pixel-by-pixel comparisons on the RBG like the L2 Norm or NCC, SSIM operates on the Lab space, taking into account metrics like brightness, contrast, and color. As such, it’s extracts information effectively when the brightness of different channels are very different (for instance, if a plate is more R dominant).
We’ll demonstrate the results for SSIM below: For the sake of notation, let A|B = (p,q) denote the positional offset of channel A with respect to channel B being a x-displacement of p and a y-displacement of q. Then, we have:
Cathedral, SSIM; R|B (3,12), G|B (2, 5)
Tobolsk, SSIM; R|B (3,6), G|B (2, 3)
Monastery, SSIM; R|B (2,3), G|B (2, -3)
Higher resolutions would demand a more exhaustive search across the grid to find proper alignment. To avoid doing this exhaustive search, we implement an image pyramid, reduce number of pixels on the image, perform an alignment search, and then increase the number of pixels, use the previous optimal alignment as a starting point, and search a small window around that starting point. We then repeat until we get to the original image size.
The results of these alignments can be seen below:
Church, SSIM; R|B: (-4,58), G|B: (4, 25)
Emir, SSIM; R|B: (40,105), G|B: (21, 50)
Harvesters, SSIM: R|B: (11, 122), G|B(14, 59)
Icon, SSIM: R|B: (23, 89), G|B: (16, 39)
Lady SSIM; R|B: (12,120), G|B (8, 57)
Onion Church SSIM; R|B (35, 108), G|B (23, 52)