draft · v0.1.226
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

4.0 BASIC IMAGE PROCESSING AND ISP⧉

This part focuses on the surprisingly serious number of steps and methods needed to go from a sensor reading to an image file stored on disk, and on the methods in use today on a typical camera, phone, or photo editor such as Adobe Lightroom. It also introduces the critical signal- and image-processing tools needed to understand them, in particular Fourier analysis. And it illustrates the range of computational photography, from objective photometric criteria to subjective beautification curves and color grading. By the end of this part you should be able to build the core of a small photo editor, a mini Lightroom with exposure, contrast, color, sharpening — or the camera-side version of the same machinery, the image signal processor that turns raw sensor data into a finished JPEG.

This part covers a lot of material. Almost every operation you associate with a photo app or a camera is in this part: making a picture brighter or punchier, fixing its color, sharpening it, blurring it, scaling it up or down, cleaning up noise, saving it small enough to send. Underneath that variety sit a handful of ideas that recur the whole way through, and we will meet them in an order designed so each one earns the next. We start with the representation itself: what the array is, what its numbers mean, and the small conventions (coordinates, channels, edges) that quietly cause most beginner bugs (this chapter). Because image code is famously easy to get subtly wrong, we then spend a short chapter on how to develop, test, and debug it, the habits that save you later. With that in hand we look at point operations (a curve applied to every pixel: exposure, contrast, tone, histograms), then tone mapping for scenes whose range outruns the display, then neighborhood operations and convolution (a pixel computed from its neighbors: blur, sharpen, gradients). Convolution opens the door to Fourier, which gives us the language for sampling, aliasing, and resampling, and the pyramids that represent an image at many scales at once. We close with the pieces a real camera bolts together: file formats and compression (how a JPEG actually works), the metrics by which images are compared, denoising and demosaicking, and finally the image signal processor, the fixed pipeline from raw sensor data to a finished JPEG that ties every chapter of this part into one block diagram.

One thread runs through all of it, and it is worth stating up front because it will save you grief: the same array of numbers can mean different things, depending on how it is encoded. Do the physics, combining exposures, white balance, and blurring, on linear values that are proportional to light; do the display and the compression on gamma-encoded values tuned to perception; and reach for log when the range is enormous. Knowing which of these a given operation wants is half of getting image processing right.

Contents of this part

▸ full collapsible outline of this part