5.0 BASIC IMAGE PROCESSING AND ISP🔗⧉

This part focuses on the steps and methods needed to go from a sensor reading to an image file stored on disk, and on the methods in use today on a typical camera, phone, or photo editor such as Adobe Lightroom. It also introduces the critical signal- and image-processing tools needed to understand them, in particular Fourier analysis. It also illustrates the range of computational photography, from objectivei photometric criteria to subjective beautification curves and color grading. By the end of this part you should be able to build the core of a small photo editor, a mini Lightroom with exposure, contrast, color, sharpening, or the camera-side version of the same machinery, the image signal processor that turns raw sensor data into a finished JPEG.

One thread runs through all of it, and it is worth stating up front because it will save you grief: the same array of numbers can mean different things, depending on how it is encoded. Do the physics, combining exposures, white balance, and blurring, on linear values that are proportional to light; do the display and the compression on gamma-encoded values tuned to perception; and reach for log when the range is enormous. Knowing which of these a given operation wants is half of getting image processing right.

Contents of this part

• 5.1 Image representation
  • 5.1.1 An image is an array
  • 5.1.2 Float vs 8-bit vs more bits
  • 5.1.3 Stochastic Quantization and Dithering
  • 5.1.4 How the array sits in memory
  • 5.1.5 Pixel coordinate conventions
  • 5.1.6 Alpha and extra channels
  • 5.1.7 Video and more dimensions
  • 5.1.8 Pixel lookups and boundaries
  • 5.1.9 Beyond the pixels: basic metadata and EXIF
  • 5.1.10 Three kinds of operation
• 5.2 What the numbers mean
  • 5.2.1 What the numbers actually mean: encoding
  • 5.2.2 What the numbers actually mean: color spaces
• 5.3 Developing, Testing and Debugging
  • 5.3.1 The workflow: build, run, look at the picture
  • 5.3.2 Principles
  • 5.3.3 Test on inputs you can verify by hand
  • 5.3.4 Finding good real test inputs
  • 5.3.5 Per-algorithm debug recipes
  • 5.3.6 Crashes and bounds
  • 5.3.7 Vibe coding: writing image code with an LLM
• 5.4 Point operations
  • 5.4.1 Three kinds of operation
  • 5.4.2 The point-operation curve
  • 5.4.3 Exposure: a multiply in linear light
  • 5.4.4 Brightness vs. exposure
  • 5.4.5 Contrast: steepen about a pivot
  • 5.4.6 Black point, white point, and levels
  • 5.4.7 The general tone curve
  • 5.4.8 Basic color enhancement: saturation and vibrance
  • 5.4.9 Converting to black and white
  • 5.4.10 Lookup tables
  • 5.4.11 Video color grading
  • 5.4.12 Local point operations
• 5.5 Histograms
  • 5.5.1 the histogram
  • 5.5.2 the histogram depends on the encoding space
  • 5.5.3 a histogram is a sampled estimate
  • 5.5.4 histogram equalization
  • 5.5.5 histogram matching
  • 5.5.6 constraining the slope: Ward's histogram-based tone mapping
• 5.6 Global tone mapping
  • 5.6.1 The dynamic-range problem
  • 5.6.2 Two families: global and local
  • 5.6.3 Global tone mapping: one curve for everything
  • 5.6.4 Histogram equalization and Ward's bound
  • 5.6.5 Local tone mapping and the halo problem: why naive local fails
  • 5.6.6 Why log space pays off
  • 5.6.7 The analog ancestor: the Zone System
  • 5.6.8 HDR on every screen: the gain map
  • 5.6.9 Where this is going
• 5.7 Beauty curves: the camera "look"
  • 5.7.1 Why scene-linear looks dull
  • 5.7.2 The shape of the curve
  • 5.7.3 Highlights: roll off to white, not to a color
  • 5.7.4 Saturation, and the rest of the "look"
  • 5.7.5 The discipline
• 5.8 Neighborhood operations and convolution
  • 5.8.1 Motivation: blur and sharpen
  • 5.8.2 Convolution 101
  • 5.8.3 The flip: where-from vs. where-to
  • 5.8.4 Properties: the impulse, normalization, symmetry, commutativity
  • 5.8.5 Nitty-gritty: finite images
  • 5.8.6 A blur zoo
  • 5.8.7 Separability
  • 5.8.8 Gradient and oriented filters
  • 5.8.9 Is imaging blur usually a shift-invariant convolution?
  • 5.8.10 Where this goes next
• 5.9 Sharpening
  • 5.9.1 Linear sharpening
  • 5.9.2 Why it's called "unsharp mask"
  • 5.9.3 Sharpen in gamma, linear, or log?
  • 5.9.4 Non-linear sharpening
• 5.10 Fourier
  • 5.10.1 Images as vectors in a high-dimensional space
  • 5.10.2 why Fourier
  • 5.10.3 Definition: one coefficient per wave
  • 5.10.4 Sines are the eigenvectors of convolution
  • 5.10.5 The 2-pixel example
  • 5.10.6 Reading an image's Fourier transform
  • 5.10.7 A small bestiary of transforms
  • 5.10.8 The fine print: two limitations of Fourier
  • 5.10.9 Windowing: the price of measuring a real spectrum
  • 5.10.10 Application preview: can we deblur, given the blur?
• 5.11 Sampling, downsampling, and aliasing
  • 5.11.1 Aliasing: a high frequency in disguise
  • 5.11.2 Nyquist and the sampling theorem
  • 5.11.3 In two dimensions, aliasing has a direction
  • 5.11.4 Sampling in the frequency domain: spectral replicas
  • 5.11.5 Seeing it: sampling and pre-filtering, hands on
  • 5.11.6 The ideal reconstruction filter: sinc, and why it is unreachable
• 5.12 Resampling and upsampling
  • 5.12.1 Domain operations: moving pixels around
  • 5.12.2 Start with scaling up
  • 5.12.3 The naive idea, and why it leaves gaps
  • 5.12.4 Loop over the output, use the inverse transform
  • 5.12.5 Nearest neighbor
  • 5.12.6 Linear interpolation, in 1-D first
  • 5.12.7 Bilinear, in 2-D
  • 5.12.8 The convolution perspective
  • 5.12.9 Better kernels: bicubic and Lanczos
  • 5.12.10 Upsampling vs. downsampling: scale the kernel
  • 5.12.11 Prefiltering when the transform isn't a clean scale
  • 5.12.12 Beyond Nyquist-limited reconstruction
  • 5.12.13 Where this goes next
• 5.13 Pyramids and wavelets
  • 5.13.1 Halfway between space and frequency
  • 5.13.2 The idea: process an image at many resolutions
  • 5.13.3 The Gaussian pyramid
  • 5.13.4 The Laplacian pyramid
  • 5.13.5 Reconstruction: an exact encoder and decoder
  • 5.13.6 Each level is a frequency band
  • 5.13.7 What the bands look like on a real image
  • 5.13.8 Candid limitations
  • 5.13.9 Cousins: wavelets and steerable pyramids
  • 5.13.10 Pyramid blending
  • 5.13.11 Other applications
  • 5.13.12 Multiresolution as a recurring theme
• 5.14 Image metrics
  • 5.14.1 Full-reference vs. no-reference
  • 5.14.2 why not just L2?
  • 5.14.3 PSNR
  • 5.14.4 SSIM
  • 5.14.5 VDP and HDR-VDP
  • 5.14.6 learned metrics
  • 5.14.7 The right metric depends on the task
• 5.15 Denoising basics
  • 5.15.1 what is noise?
  • 5.15.2 denoising by averaging multiple shots
  • 5.15.3 denoising from a single image
  • 5.15.4 Spatial averaging and its limits
  • 5.15.5 The bilateral filter: averaging by affinity
  • 5.15.6 Denoising in the pyramid: coring
  • 5.15.7 Denoise color more than brightness
  • 5.15.8 noise estimation
  • 5.15.9 The limits of denoising
• 5.16 Demosaicking
  • 5.16.1 Reminder: the Bayer mosaic
  • 5.16.2 The task: full RGB at every pixel
  • 5.16.3 The naive approach: interpolate each channel on its own
  • 5.16.4 Why naive interpolation zippers: averaging across an edge
  • 5.16.5 Doing better: edge-directed interpolation
  • 5.16.6 The harder half: red and blue, and color fringing
  • 5.16.7 Green-based demosaicking: interpolate the color difference
  • 5.16.8 Classic (non-learning) demosaicking: the general strategy
  • 5.16.9 Related: the optical anti-aliasing filter
  • 5.16.10 Quad-Bayer sensors: remosaic before demosaicking
  • 5.16.11 Fuji X-Trans: a larger, irregular CFA
  • 5.16.12 Beyond hand-tuned: joint denoising and learned demosaicking
  • 5.16.13 Cross-reference: other ways to sense color
  • 5.16.14 Where this sits in the pipeline
• 5.17 Auto-exposure and auto white balance
  • 5.17.1 Auto-exposure: metering
  • 5.17.2 white balance and color constancy
  • 5.17.3 Automatic white balance
  • 5.17.4 The limits of white balance, and CRI
• 5.18 File formats and compression
  • 5.18.1 The big picture: none, lossless, lossy
  • 5.18.2 Data versus metadata: EXIF
  • 5.18.3 PNG: the format we read from
  • 5.18.4 JPEG: compression by perception
  • 5.18.5 RAW files: before the cooking
  • 5.18.6 HDR formats: more than 8 bits
  • 5.18.7 Modern formats
  • 5.18.8 Other formats: TIFF and GIF
• 5.19 Recap ISP, non-destructive editing:
  • 5.19.1 A basic ISP
  • 5.19.2 Pipeline design and tuning
  • 5.19.3 The ISP, evolving: traditional → learned → generative
  • 5.19.4 Recap 2: non-destructive editing

▸ full collapsible outline of this part