At first, the digital revolution merely replaced film by digital sensors more amenable to quick snapshots, instant gratification and sharing. But it did more than that : it created the opportunity to add arbitrary computation between the photons in a scene and the final image.

This allows us to do really cool stuff : manage scenes where contrast is excessive, computationally remove blur, combine images into virtual wide angle panoramas, and even capture the full array of light to do stuff like refocusing after the fact and extract 3D.

[results-montage figure — HDR tone mapping (Durand et al.), refocus-after-capture (Ng et al.), motion deblur (Levin et al.), panorama (problem set)]

In fact, even a simple snapshot requires serious computation because each pixel usually captures only one of the 3 channels and the other two need to be reconstructed [demosaick-snapshot figure]. Cell phones these days take much better images than they have a right to because of computational photography.

Plus generative AI is poised to further revolutionize photography and image making with its ability to learn the space of all possible images in a manner reminiscent of a Borges novella (cite *The Library of Babel* and https://arxiv.org/abs/2310.01425)

• the book is organized **cross-cuttingly**: a chapter is sometimes a **tool** (convolution, the bilateral filter), sometimes a **big idea** (priors, the light field), and the major themes deliberately **recur** across chapters — the same lesson seen from several angles. The redundancy is intentional; cross-references thread the pieces together.

• **hammers and nails**: we deliberately sought a **balance between "hammers and nails"** — between **tools/techniques** (the hammers: convolution, the bilateral filter, optimization, priors) and **applications/challenges** (the nails: HDR, panoramas, deblurring, refocusing). Rather than a purely tool-first or purely application-first table of contents, the book's structure is **organized by a combination of tools and applications/challenges**, alternating between the two so each motivates the other.

• **intuition first**: lead with the idea and a picture, then the math. The audience is a **CS undergrad**, new to imaging (what to know coming in is spelled out in *[[#Prerequisites]]* below).

• **how to read**: FUNDAMENTALS grounds the physics and perception; BASIC IMAGE PROCESSING gives the toolkit; later parts build single- and multi-image computational photography. Read mostly front-to-back, or follow the dependency graph below to chart your own path.

• **how this book came to be**:

• grew out of Frédo's **computational photography course** at MIT — its lectures, slides, and problem sets are this book's backbone.

• written **with an AI collaborator** (Claude) as a deliberate experiment in **AI-assisted authoring** — the outline, figures, drafts, and the book's own tooling were co-developed; a fitting, slightly meta nod to the generative-AI theme the book itself discusses.

• the **process** (how outline, figures, and prose are generated, verified, and kept consistent) is documented in an appendix — itself a small piece of AI methodology [forward ref: "how this book was created"].

• **chapter dependency graph**:

• *(placeholder — to create later)* a **graph of chapter dependencies** belongs here, at the end of the intro: a diagram showing which chapters rely on which, so a reader can chart their own path through the book. Generate it from the **provides / requires** graph (see [[Book Generation Process#Cross-chapter dependencies]] and each section's `- prereqs:`), once enough of the book is written for the dependencies to be stable.

• a **short, intro-level** chapter (the headline version of the appendix [[Appendices#How this book was created]]): the book is partly *about* generative AI and was partly written *with* it, so the making-of is part of the story, not a footnote.

• **where it came from**: grew out of Frédo's MIT computational photography course — ~**305k words** of auto-captured/auto-transcribed lecture transcripts, ~**114k** of slide decks, ~**42k** of problem sets (~**460k** of pre-existing teaching material) form the backbone.

• **how it was written**: drafted **with an AI collaborator** (Claude); outline/prose split with a compile step; figures are programs; the human edits triage back so re-compiling reproduces the corrections.

• **who did what**: AI for laborious-but-specified drafting, figure code, and bookkeeping; human for structure, judgment, catching subtle errors, and ownership of every claim.

• **the numbers, honestly**: ~**40.5k words** of typed prompts steered ~**735k words** of generated outline+prose (≈13–18× leverage, *not* authorship); the outline is model output too, so the earlier "2.2× human writing" framing was wrong — the real human contribution is **curation**, not keystrokes. Point to the two appendices for the full account and the under-the-hood prompts/verifiers.

• the book is **language-agnostic in its ideas** but ships in two editions — a **Python** one and a **C++** one (never one book showing both) — because the two languages serve different moments: Python to *think and prototype*, C++ to *ship and run fast*. The concepts are identical; pick the edition that matches your goal.

• **why choose which**: **Python** for **deep learning, prototyping, and convenience**; **C++** for **performance**. The gap is not small. When you write the **low-level code yourself** (e.g. your own convolution rather than a library call), the **speed differential** is dramatic: in Frédo's experience, roughly **1000× from Python to optimized C++**, and **at least another ~10× from C++ to Halide**. So the language choice is also a performance choice — see [[Performance engineering and Halide]] and [[Appendices#Programming: Python, C++, and PyTorch]].

• **Python** — the right default for *learning, prototyping, and research*:

• **pros**: **readable** — NumPy array code reads almost like the math (`out = a + 0.5*b`); **fast to write and explore**, with interactive notebooks and instant feedback; a vast **ecosystem** (NumPy, SciPy, Pillow / OpenCV, scikit-image, Matplotlib, and the whole deep-learning stack — PyTorch, JAX); automatic memory; the lingua franca of research and ML.

• **cons**: **slow at the pixel level** — a naive per-pixel Python loop is orders of magnitude slower than C, so you must **vectorize** (push the loop down into NumPy's C core) or it crawls; the **GIL** limits real CPU threading; packaging / deployment is awkward; dynamic typing lets type bugs hide until run time; little control over memory layout or exact performance.

• bottom line: the right default for **learning, prototyping, and research** — and, through NumPy / PyTorch, fast *enough* for most of what we do here.

• **C++** — the right default for *performance and production*:

• **pros**: **fast and predictable** — close to the metal, with control over memory layout, cache behaviour, and SIMD; what real **camera ISPs, real-time pipelines, and shipped products** are written in; static types catch errors at compile time; deploys to phones and embedded hardware.

• **cons**: **verbose and slower to write**; **manual memory management**, a rich source of bugs (out-of-bounds, leaks, use-after-free); a heavier **build / toolchain** story; far less interactive (edit–compile–run, not edit–run). The productivity tax is real.

• bottom line: the right default for **performance and production** — once the algorithm is settled and it must run fast, on device, at scale.

• **Halide** — a **domain-specific language** (embedded in C++ / Python) for high-performance image processing: you write the **algorithm** (what each pixel computes) **separately** from the **schedule** (how to tile, vectorize, parallelize, and fuse it across the memory hierarchy).

• **pros**: near-hand-tuned speed **without** hand-writing the tangled, hardware-specific loops; the *same* algorithm retargets to CPU / GPU / DSP by swapping the schedule; the basis of real camera stacks.

• **cons**: another language and mental model to learn, best reserved for the **hot paths** you have already identified — premature scheduling is its own trap. Covered properly in [[Performance engineering and Halide]].

• **MATLAB (historical note)**: for years *the* platform for this work — a strong numerical/linear-algebra library and an IDE that handled **images natively** (an image is a matrix; `imread`/`imshow`/slicing; the Image Processing Toolbox); much of the literature was prototyped in it. **Largely displaced by Python** (NumPy fills the same array niche, free) and especially **PyTorch** (autodiff + GPU + deep learning, which MATLAB never matched). Worth knowing because older papers/code are in it, and its array-thinking carries over to NumPy.

• **Libraries (implement-from-scratch, then build on)**: the book has you **implement core algorithms from scratch** to understand them (the `imageops` reference), but to *build* / dive into advanced projects, stand on mature libraries — **Python**: Pillow (PIL), NumPy/SciPy, OpenCV, scikit-image, PyTorch (also a GPU-array + autodiff engine); **C++**: OpenCV (de-facto standard), Eigen (linear algebra), LibRaw (raw decode), Halide (hot paths).

• **side bar — Photoshop 1.0, in 128,000 lines (a language lesson from 1990)**: Adobe (via the **Computer History Museum**, 2013) released the source of **Photoshop 1.0.1** — **≈128,000 lines across 179 files**, written for the Macintosh by **Thomas Knoll** (its sole engineer for v1 — a computer-vision PhD student whose 1987 image-display hack became the app). The language split *is* this section's lesson, 35 years early: **≈75 % Pascal** (the productive high-level language) and **≈15 % hand-written 68000 assembly** for the speed-critical inner loops — *most of it readable, the hot pixels hand-tuned* — exactly the Python-vs-C++/Halide trade-off (a high-level language for the bulk, machine code where it counts; what **Halide** now automates). The code is famously *mostly uncommented but well-structured*. It's now studied as a cultural artifact: the **"Das digitale Bild"** project (DFG / LMU Munich) runs an ongoing **Critical-Code-Studies** *"reading the source code of Photoshop"* — code as a text to be close-read. **For scale:** today's Photoshop is reportedly **millions** of lines of C++ (Frédo's estimate ~**10 M**); **Adobe Camera Raw (ACR)** and **Lightroom** share the same Camera Raw engine, and a typical camera **ISP** is a large proprietary C/C++ codebase — exact modern sizes mostly **not public** [⚠️ confirm the ~10 M / current ACR/Lightroom/ISP figures]. (→ the hand-tuned-inner-loop angle returns in [[Performance engineering and Halide]].)

• **Vibe coding (LLM-assisted)**:

• writing image code with an **LLM** ("vibe coding") is now part of the toolkit: great for **boilerplate, I/O, visualization, and explaining errors**, and a quick way to draft a function or a test.

• **but** image code is **numerically and perceptually subtle** — "looks plausible" is not "correct" — so keep the model on a **short leash** for filtering, resampling, color, and antialiasing, and always **verify against a hand-checkable input** (a constant, an impulse, a half-plane) before trusting it. Treat generated image code as *guilty until tested* (see [[Developing, Testing and Debugging]]). AI coding sometimes **fumbles the trivial** even when it nails the hard parts — see the cross-fade anecdote in [[Basic image processing and ISP#Vibe coding: writing image code with an LLM]].

• **PS0 — Environment & C++ basics**: toolchain setup, a minimal `Image` class, image I/O. (→ [[Developing, Testing and Debugging]])

• **PS1 — Image class, point operations, color**: pixel access, brightness/contrast, luminance–chrominance (YUV), gamma, white balance, the Spanish-Castle illusion. (→ [[Image representation]], [[Point operations]], [[Perceptual color and trichromatic vision]])

• **PS2 — Convolution & the bilateral filter**: box/Gaussian blur, gradients, sharpening, and bilateral denoising (with a YUV variant). (→ [[Neighborhood operations and convolution]], [[Bilateral filtering]])

• **PS3 — Denoising & demosaicking**: align-and-average a burst, ISO/variance; Bayer demosaicking (green-first, edge-based, color-difference) and Prokudin-Gorsky channel alignment. (→ [[Denoising basics]], [[Demosaicking]])

• **PS4 — HDR**: merge an exposure bracket into a radiance map and tone-map it. (→ [[HDR merging]], [[Global tone mapping]])

• **PS5 — Resampling, warping & morphing**: nearest/bilinear/bicubic/Lanczos resampling; Beier–Neely segment warps; a full Beier–Neely morph. (→ [[Warping]], [[Morphing]])

• **PS6 — Homographies & manual panoramas**: homogeneous coordinates, warp by a homography, solve $H$ from four correspondences, stitch a planar mosaic. (→ [[Manual panorama stitching from multiple views]])

• **PS7 — Automatic panoramas**: Harris corners (structure tensor), patch descriptors, the second-nearest-neighbour ratio test, RANSAC, and feathered / two-scale blending. (→ [[Automatic panorama stitching from multiple views and feature matching]])

• **PS8 — Non-photorealistic rendering**: paintbrush splatting; single-scale, two-scale, and gradient-oriented painterly rendering. (→ [[Non-photorealistic rendering]])

• **PS9 — Make-your-own / video & ethics**: a self-proposed project (optical-flow retiming, phase-based video magnification, …) plus the ethics deliverable; its open menu of advanced topics seeds many [[Computational optics and coded imaging]] sections. (→ [[Optical flow]], [[Motion and video magnification|Video magnification]])

• chapter intro: a working model of light — its **color** (spectrum), its **interaction with surfaces**, and its **energy** (radiometry); closes with the **plenoptic function** that ties the pipeline together

• the perceptual machinery the visual system runs *on top of* color: adapting to level, discounting the illuminant to report surface reflectance (constancy), working in ratios (contrast), and resolving detail unevenly across spatial/temporal frequency (the CSFs) — the properties compression, tone mapping, and display design optimize for.

• **the eye evolved as a sequence of small improvements**, and it recapitulates the optics of this book — from "no image" to a focused one. Following Nilsson's stages (Land & Nilsson, *Animal Eyes*):

• **a flat patch of photoreceptors** (an *eyespot*): just **photosites**, **no image formation** — it tells light from dark and, crudely, which side has *more* light (phototaxis), but cannot form a picture. This is the bare **sensor** with no optics.

• **a concavity / cup**: fold the patch into a **pit** and the rim's shading gives **directional** sensitivity — the deeper the cup, the better you can tell *where* light comes from.

• **a pinhole**: close the cup almost shut → a **pinhole eye**, a real (if dim) **image** with no lens at all (the **nautilus** still uses one) — exactly the camera obscura of [[#Pinhole image formation and linear perspective]].

• **a lens**: fill the aperture with a **refractive lens** (often a graded-index sphere) → a **bright, focused** image, independently evolved in fish, cephalopods, and vertebrates. Add a **cornea, iris, and accommodation** and you have the human eye.

• so the human eye is *one* solution; evolution reached the same goal by very different routes (below).

• **a tour of those other solutions** — a useful contrast to the human eye and to camera design:

• **compound eyes** (insects): many **ommatidia** → wide field of view and fast temporal response, but low spatial resolution

• **more cone types**: birds / reptiles are **tetrachromatic** (a 4th cone, into the UV); the **mantis shrimp** has ~12–16 photoreceptor classes (yet poor color *discrimination* — it seems to recognize rather than compare) — the **divergence of opsins** across animal groups is the color sidebar in [[#Color]]

• **tapetum lucidum** (cats, deer): a reflective layer behind the retina that boosts night vision (and causes eye-shine)

• **polarization vision** (cephalopods, many insects): they see light's **polarization** — a channel we're blind to

• **other eye designs**: **pinhole** (nautilus), **mirror eyes** (scallops — image formed by a concave mirror, not a lens), and the front-facing (**stereo**) vs side-facing (**field-of-view**) placement trade-off

• further reading: Land & Nilsson, *Animal Eyes*; the "Evolution of Eyes" chapter in *Vision* (Cambridge); a friendly overview at phos.co.uk, "The Evolution of Sight".

• sensor alone would capture a diffuse light. We need to select which rays from which part of the scene contribute to which pixel.

• chapter intro: from the **pinhole** (sharp but dim) to a **lens** (bright but with one focus plane) — refraction at curved surfaces, the thin-lens linearization, and the depth-of-field that the finite aperture forces on us

• chapter intro: the unifying idea behind sensing — **measurement = integrating a slice of the plenoptic function** — then the physical sensor that does it (photosites, CCD/CMOS) and the **noise** that limits it

• sidebar — **photography as objective measurement**: the medium's long arc bends toward objective measurement (film/print = a *performance*; → measurement). The **digital sensor** is the far end: because each photosite *counts photons*, a **raw** image is ≈ a **calibrated radiometric measurement** — a physical map of scene radiance, in real units, of one snapshot of time — not just a rendering. This is what makes HDR / depth / deblur possible: the pixel numbers must *mean* something physical. Ties to the intro's **generation → measurement → generation** arc (→ [[01 Intro]]).

• **defocus / circle of confusion**: a point off the focus plane images to a point that is not on the sensor, so the aperture cone meets the sensor as a blur disk — the **circle of confusion**. A point still counts as sharp while its disk stays within an acceptable c. [CoC figure]

• the blur disk grows with the **aperture** (wider → bigger disk) and with the **object's distance from the focus plane** (zero exactly at focus); focal length and focus distance enter too. [defocus-vs-parameters figure]

• **depth of field** = the *range* of object distances whose blur disk stays within c; the **near** and **far** limits are where the blur just reaches c. [DoF near/far figure — each cone carried out to the CoC at the sensor]

• **deriving the limits**: on the image side a near-limit object images *behind* the sensor and a far-limit object *in front* of it; **similar triangles** relate the aperture D, that convergence distance, and the circle of confusion c — and together with the conjugate relations (1/f = 1/s + 1/v) they fix the near (front) and far (back) object-distance limits. [near-limit and far-limit similar-triangle figures]

• **hyperfocal distance** H = f²/(N·c): focus there and everything from **H/2 to infinity** is acceptably sharp — the focus setting that maximizes DoF (focusing at ∞ instead wastes the H/2…H zone). [hyperfocal figure]

• DoF **grows** with a smaller aperture (larger N), a shorter focal length, and a greater focus distance — these are the paraxial (linearized) formulas; the accurate model is in the Optics part. [DoF-vs-parameters figure]

• **the surprising invariance** (the lecture's flagged "important conclusion"): for a **fixed framing** (same subject size) and a **fixed f-number**, the depth of field in **object space** is essentially **independent of focal length** — switching 28 mm → 100 mm forces you to step back, and the larger physical aperture cancels the longer focal length. So "long lenses have shallow DoF" is really about **magnification**, not focal length per se. (The *background* still looks more blurred with the long lens — that's magnification of the out-of-focus disk, not a DoF change.) [same-framing/same-N → same DoF figure]

• ⚠️ **but the invariance is only *first-order* — the background gives the focal length away.** "Same framing + same f-number → same DoF" is the **small-defocus (paraxial)** result: it holds for the **near/far acceptable-sharpness limits straddling the subject**, but **not** for how hard a *distant* background is thrown out of focus. **Derivation** (thin lens; blur-disk diameter on the sensor): a point at distance $s_b$, with the lens focused at $s_f$, images to a circle of confusion $c \approx \dfrac{f^2}{N}\,\dfrac{|s_b-s_f|}{s_b\,s_f}$. **Fix the framing** — subject magnification $m=f/s_f$ held constant, so $s_f=f/m$ — and **fix $N$**, with the background a gap $\Delta=s_b-s_f$ behind the subject:

$$c(f)=\frac{m^2\Delta/N}{\,1+ m\Delta/f\,}\ \xrightarrow[f\to\infty]{}\ \frac{m^2\Delta}{N}.$$

The subject-zone DoF is ~independent of $f$ (the leading $f$ cancels — the **first-order invariance**), but the **background blur grows monotonically with focal length**, saturating at $m^2\Delta/N$. Two shots matched in framing *and* aperture render the **subject** identically, yet a **200 mm melts a far background far more** than a 35 mm: the correction term is order $m\Delta/f$ — negligible for a background near the focus plane (invariance looks exact), but dominant once the background is many focus-distances away. This is exactly why portrait shooters reach for long lenses for "background separation" even though the DoF on the *face* is unchanged. **Full derivation + plots** of $c$ vs. background distance for 35 / 85 / 200 mm (fixed framing & $N$) — the curves coincide near the focus plane and fan apart far from it. [`fig-dof-background-blur`]

• **object-space picture & a caveat**: the simplest view is a **double cone** in front of the lens whose waist sits on the focus plane — it gives the blur for **one** scene point. But **defocus is *not* a spatially-invariant convolution**: the circle of confusion depends on each point's **scene depth** (not its pixel location), and at depth discontinuities **occlusion** matters — which is why naive "just blur the image" fakes look wrong, and why light-field / depth-aware methods (Advanced) do better. [object-space double cone; depth-dependent-blur + occlusion figure]

• **sensor size scales DoF**: shrink the sensor (keeping FOV) and both the circle of confusion and the aperture shrink, so DoF grows **linearly** with sensor size — why phone cameras have huge DoF (and resort to *computational* background blur), and why **macro is easier on small sensors** (a scaled-down camera photographs a scaled-down world — modulo diffraction)

• 🖱️ **Interactive (web edition):** a live **macro depth-of-field simulator** — a to-scale thin-lens diagram, a lens-supersampled 3D photo (real bokeh), and a circle-of-confusion-vs-distance plot all driven by one shared optics model (focal length, f-number, sensor, focus distance, subject–background gap). A 3D bug (beetle / bee / ladybug) over a flower makes the macro lesson vivid: focus lands on the front of the face and the rest of the body, antennae, and background all fall out of focus. [fig-depth-of-field-sim; interactive] *(bug & flower 3D models generated with Meshy AI — acknowledge in the caption)*

• **blur from subject motion**: $b_{px}\approx f_{px}\,v\Delta t/D$ — a straight streak = image velocity × exposure (constant-velocity linearization); only cross-axis motion blurs

• **blur from camera shake**: translation $b\approx f_{px}\Delta_c/D$ (depth-dependent); rotation yaw/pitch $b\approx f_{px}\theta$ (**depth-independent**, small-angle); roll = spin about centre, grows with radius

• **which dominates**: ratio $\theta D/\Delta_c$ → rotation wins at normal distance, translation only up close (macro); rotation blur is **spatially non-uniform** so one kernel can't undo it (Whyte); IMUs measure the rotation to deblur (Joshi)

• **hand-holding rule** $\Delta t\lesssim 1/f$ s (×crop; ~2× faster for dense sensors); **image stabilization** cancels camera *rotation* (buys 3–5 stops) but not *subject* motion

• 🖱️ **Interactive:** motion-blur calculator — 3-D scene (Meshy walking human / car), temporal-supersampled blur, world+screen readout by source, motion-vector & flow-colour overlays, hover pixel scale [fig-motion-blur-calculator; interactive]

• four ways to turn a monochrome sensor into a color one — **multiplex** the three (or more) measurements across some axis:

• **in time**: shoot R, G, B frames sequentially through filters — Maxwell's first color photograph (1861), **Prokudin-Gorskii**, flatbed scanners, astronomy filter wheels. Cheap and full-resolution, but **fails on motion** (the classic color fringes on moving water).

• **in space (the dominant choice)**: a **color filter array (CFA)** — the **Bayer mosaic** (2 green : 1 red : 1 blue) — one color per photosite, then **demosaick** to fill the rest; the eye does the same with its interleaved cone mosaic. Variants: Fuji's **X-Trans** (a larger, less-periodic tile to fight moiré) and **complementary CMY/CYGM** CFAs (more light through, messier color). Needs an **optical anti-aliasing (low-pass) filter** to tame high-frequency color artifacts, and suffers some channel **crosstalk**.

• **beam-splitter (prism)**: a **dichroic prism** splits the rays onto **three sensors** — **3-CCD / 3-chip**, the classic **broadcast video camera** — wasting no photons at full per-channel resolution, but bulky, costly, and hard to align.

• **in depth (stacked)**: stack wavelength-selective layers so one location senses all three — **Foveon** (silicon absorbs longer wavelengths deeper; used in **Sigma** cameras), echoing **Kodachrome**'s stacked dye layers. Full color at every pixel (**no demosaicking**), but trickier color separation and more chroma noise.

• **spectral / Lippmann**: record the standing-wave **interference** of the full spectrum (Lippmann, 1908 Nobel; a precursor to holography) — true spectral capture, not just 3 numbers.

• **hybrid**: combine these (e.g. spatial + temporal in video).

• this is *analysis* (sensing color) — the mirror of color *synthesis* (reproduction, above); the **demosaicking** algorithm itself is deferred to Basic Image Processing (which is why this is just the *sensing* half)

• **trichromatic (perceptual) vs spectral (physical) capture**: almost all imaging systems aim to capture **perceptual, trichromatic color** — three numbers that reproduce what a *human* would see (RGB ≈ the cones), so metamers are, by design, indistinguishable. **Multispectral / hyperspectral** imaging instead samples the **physical spectrum** in many narrow bands (tens to hundreds), keeping distinctions the eye throws away — for remote sensing, agriculture, art conservation, and machine vision, where the *material*, not the *appearance*, is what matters (and metamers must be told apart). Lippmann (above) is the limiting case: the full continuous spectrum.

• the noise **sources** (their variances add):

• **photon / shot noise** — the Poisson statistics of *counting photons*: variance = mean, so σ = √N. Fundamental (it's the light itself), dominates the midtones/highlights; averaging N independent frames cuts it by √N.

• **read noise** — added by the amplifier / ADC on readout; signal-independent, dominates the **shadows** and sets the noise floor (hence dynamic range).

• **thermal / dark current** — electrons freed by heat, independent of light; grows with exposure time and temperature (long exposures & astrophotography → sensor cooling, dark-frame subtraction).

• **fixed-pattern / pattern noise** — per-pixel gain/offset non-uniformity (PRNU/DSNU), hot/stuck pixels, banding; *structured*, so it reads as worse than random noise of equal magnitude, and is removed by calibration (flat / dark fields).

• 💡 **Big lesson — in a *linear* image, noise variance is an *affine* function of brightness.** The variances add: shot noise gives a term **proportional to the signal** (Poisson, variance = mean, scaled by the gain) and read noise a **constant** floor, so the total is **variance ≈ gain·signal + read²** — a straight line in brightness. You can *measure* it: take an aligned **burst** of a static scene and, per pixel, plot the variance across frames against the mean — the points trace that line, slope = photon gain, intercept = the read-noise floor. And a single pixel's value over many frames is **≈ Gaussian**, which is what licenses additive-Gaussian noise models. [`fig-noise-affine` — per-pixel variance vs brightness, measured from a 50-frame ISO-3200 burst; `fig-noise-gaussian-pixels` — per-pixel histograms] → register [[Big Lessons]]

• 💡 **Big lesson — noise is *clipped* at black and white, so near the extremes it is *not* zero-mean.** A recorded value is clamped to [0, max]: near black the negative half of the noise is cut off (frames pile up at 0), near white the positive half is. So at the extremes the noise distribution is **asymmetric** and its mean is pushed **inward**. The consequence for **denoising**: any naive average/smoothing returns that biased mean, so it makes **shadows come out too bright and highlights too dark** — a bias every denoiser has to correct for (carried forward to [[#Denoising basics]] in BASIC). [`fig-noise-truncation` — a near-black pixel pinned at 0 a third of the time, its average biased bright; from the ISO-3200 burst] → register [[Big Lessons]]

• **the key intuition — it's about ratios (SNR)**: shot noise is **Poisson**, so σ = √N grows with brightness — measured noise is actually **higher in the highlights**. Yet noise *looks* worst in the **shadows**, because perception cares about the **signal-to-noise ratio** SNR = N/√N = √N, which is worst where N is small. "Ratios are all that matters" — the same reason we encode in gamma/log, and the motivation for ETTR (below — Photography 101). [std-dev map vs SNR map figure]

• **dynamic range** = the brightest recordable signal (sensor **saturation / full-well**) ÷ the **noise floor** (read noise in the shadows); it sets how much of a high-contrast scene fits in a *single* exposure → the motivation for **HDR** (Multiple exposure)

• **HDR (high dynamic range)**, sometimes called **wide dynamic range (WDR)**, is a deliberately **fuzzy** term — flag this. It gets used for at least four different things: the *scene* (a high-contrast world), the *capture* (multi-exposure / a high-DR sensor), the *file/encoding* (float or ≥10-bit radiance, e.g. OpenEXR, HDR10), and the *display* (an HDR monitor) — and, loosely, for the **tone-mapped "HDR look."** There is no single sharp definition; say which sense you mean. (Developed in [[Multiple exposure imaging]].)

• 💡 **Big lesson (L2.48) — dynamic range is set by full-well capacity over the noise floor**: a single exposure records from a **top** (the photosite's **full-well capacity** — where it clips to white) down to a **bottom** (the **noise floor** — read noise in the shadows); their **ratio is the dynamic range** (quoted in stops). You **widen** it with a **bigger well** (larger photosites — why a full-frame sensor out-ranges a phone) or a **lower floor** (cooling, lower read noise), and you **beat** it altogether by merging **multiple exposures** (HDR). The capture-side companion to **L2.49**: what bounds a shot is this *range*, not the bit depth. → register [[Big Lessons]]

• **examples, in stops** (the figure): a color **slide** is narrow (~5–6 stops), a reflective **print** ~6–7, **film negative** wide (~12–13 latitude), a **phone** sensor ~10–12, a **full-frame** sensor ~14; the **human eye** is ~10–14 stops *instantaneously* but ~**20+** once **adaptation** is allowed — and some **animals** do better in their niche; an outdoor sun-and-shadow scene can exceed **20 stops**, which is why no single capture holds it [fig-dynamic-range-comparison]

• **see it — recover an underexposed shot, full-frame vs phone**: shoot the *same* underexposed scene as **raw** on a **full-frame** camera and on a **phone**, then **push the exposure up** in software (e.g. +3–4 stops). The full-frame frame cleans up (deep full-well + low read-noise floor → wide dynamic range), while the phone frame breaks down into **shadow noise and banding** as the lift amplifies its read-noise floor — a direct, visible demonstration that **dynamic range scales with photosite / sensor size**, and exactly why phones lean on **burst / HDR+** (Multiple exposure) rather than a single exposure. [fig-underexposure-recovery-ff-vs-phone; photo — paired Fredo full-frame DNG + phone raw] *(also a coding exercise — see [[Exercises & Experiments]])*

• 💡 **Big lesson (L2.49) — quantization is rarely the real problem**: with enough bits and a sane encoding, it's **noise and dynamic range** (this section) that bite — not the number of levels. The one exception is **gamma / banding in shadows**, which is exactly why gamma encoding exists (allocate codes perceptually); so "rarely" ≠ "never". Reminded again in BASIC → Image representation → *Float vs 8-bit*.

• chapter intro: the last link in the chain. The range of displays (phone, laptop, TV, projector, print) and how their characteristics — size and viewing distance, resolution, dynamic range, gamut, and viewing environment — shape both the experience and the processing. This chapter also carries color management: the ICC workflow and the industry standards that keep color consistent across capture, editing, and display. Forward-reference to [[Integral and immersive imaging]] for near-eye and immersive displays.

• chapter intro: having built the physics, we now sit behind a real camera — its **controls, modes, and guts** — and see how phones reach the same goal by a wholly different (computational) route

• chapter intro: what a focal length is *for*, fast vs slow, prime vs zoom, image stabilization, keeping the glass clean, and the filters that change the captured light.

• **the two classic passive AF families**:

• **contrast-detection (CDAF)**: drive the lens until image **contrast/sharpness is maximal** — accurate but has to **hunt** (it doesn't know which way is "more in focus", only that it overshot); historically slow, common on early mirrorless / compacts

• **phase-detection (PDAF)**: look at the scene through **two opposite edges of the lens** (sub-apertures) → two slightly shifted images; the **phase difference gives the direction *and* amount** of defocus in **one shot**, so the lens moves straight to focus (open-loop, no hunt). Classic SLR mechanism (a dedicated AF sensor under the mirror); it is literally a **tiny stereo measurement**

• **on-sensor PDAF / dual-pixel**: mirrorless bodies put phase detection *on the imaging sensor* — masked AF pixels, or **dual-pixel** designs that split each photosite into two halves that see opposite sub-apertures → phase detection at (nearly) every pixel, the best of both worlds (used for Pixel's portrait-mode depth too — a literal micro-stereo pair)

• **depth-from-defocus AF (Panasonic DFD)**: a third route — model how the lens's blur (PSF) changes with focus and read the **defocus direction and amount** from **two frames at slightly different focus**, giving a PDAF-like one-step cue from a plain **contrast** sensor (no phase pixels); rides on CDAF, needs a **per-lens blur profile** (full treatment → Optics, *Focus*)

• **active AF** (for completeness / history): time-of-flight ultrasonic (old Polaroid sonar) and IR rangefinding — work in the dark but limited; distinct from an **AF-assist lamp** that merely adds contrast for passive AF

• **focus modes**: **AF-S / one-shot** (lock once, for still subjects) vs **AF-C / AI-Servo** (track continuously, predict motion, for action); plus **MF** with focus aids (peaking, magnify — see UI)

• **focus selection & automation** — where computation now lives: single-point / zone / wide-area selection, then **subject-detection AF** — **face → eye → animal/bird/vehicle** detect that picks and **tracks** the subject (deep-learning driven; →ML). **Eye-AF** is the headline modern feature for portraits.

• **handling tricks**: **back-button focus** (decouple AF from the shutter button — focus with a thumb button, shutter only releases) and **focus-and-recompose** (lock focus on the subject, then reframe — beware the small focus-plane error at wide apertures / close range)

• chapter intro: the viewfinder and its overlays, the anatomy of a mirrorless body and of a phone, camera vs phone, the taxonomy of camera types, cameras that measure rather than photograph, and the non-imaging sensor suite.

• **frame rate & shutter angle**: video sets exposure time as a fraction of the frame interval — the **180° shutter-angle** convention (shutter ≈ ½ the frame time) gives "natural" motion blur; high frame rate → slow motion

• **rolling shutter / "jello"**: CMOS reads the sensor **row by row**, so fast motion (or a fast pan, or a spinning prop) **skews/wobbles** — a **global shutter** avoids it but is rarer/costlier (ties back to CMOS readout in Image measurements as integrals)

• **log / flat profiles + grading**: shoot a **log** (flat, low-contrast) profile to preserve dynamic range, then **color-grade** in post — the video cousin of raw + tone curve

• **codecs & bitrate**: intra- vs inter-frame compression, 8- vs 10-bit, chroma subsampling (4:2:0 vs 4:2:2), bitrate — trade file size vs editing/grading headroom

• *(kept deliberately brief — the algorithms and the full motion treatment are in **Motion/Video**)*

• **Natural illumination**: sun (hard, small) + sky (soft, huge dome) and their changing ratio; time of day (noon toplight → warm low sun; **golden hour**, **blue hour**); weather (overcast = giant diffuser; haze/aerial perspective); **sky models** (CIE distributions; Perez all-weather, Preetham daylight, Hošek–Wilkie), tuned by turbidity.

• 🖱️ **Interactive (web edition):** a 3-D **natural-illumination / physical-sky simulator** — a head/figure under a modeled sky dome; controls for sun **azimuth/elevation** and **turbidity**; a toggle shows the sky **approximated by many point lights** over the hemisphere (plus a sun key). [fig-sky-illumination-sim; interactive]

• **TTL metering**: the camera fires a **pre-flash**, meters the return **through the lens**, and sets flash power automatically — the flash analog of auto-exposure

• **sync speed & high-speed sync**: a **focal-plane** shutter only fully uncovers the sensor up to its **X-sync speed** (~1/200 s) — faster than that and a moving slit means the flash can't light the whole frame; **high-speed sync (HSS)** pulses the flash rapidly to cover the slit (at a big power cost). A **leaf shutter** (in-lens) opens fully at *any* speed → **syncs at all shutter speeds** (a real advantage for daylight fill).

• **fill flash**: in harsh/backlit light, add flash at **−1 to −2 EV** (the slides say −1.5 to −2, i.e. 3–4× below ambient) to **open the shadows** without looking flashed — it's **dynamic-range management**, not "more light"

• **bounce & off-camera**: never bare flash head-on (flat, harsh, red-eye) — **bounce** off a ceiling/wall (~45°) or use a diffuser to enlarge the source → softer light; **off-camera** flash gives directional, shaped light (→ Computational illumination; computational bounce flash, Davis SIGA 2016)

• 🖱️ **Interactive (web edition):** a live **portrait-lighting simulator** — three soft area lights (key / fill / kicker, each with azimuth, elevation, size/softness, intensity, and colour) on a 3D face, with **area-light-supersampled soft shadows** and a physiological melanin/hemoglobin skin model; a from-behind "setup" view shows the lights as studio umbrellas plus the camera, and lighting presets (butterfly, loop, Rembrandt, split, clamshell, rim) let you feel how light *placement and size* sculpt a face. [fig-portrait-lighting-sim; interactive] *(3D studio umbrella generated with Meshy AI — acknowledge in the caption)*

• most cameras run **closed firmware**: you cannot run your own code on them, so the **computational-photography ideas in this book can't be tried on a stock camera** — a real obstacle for experimentation

• the **hacks**: **CHDK** (Canon PowerShot) and **Magic Lantern** (Canon DSLRs) are community firmware add-ons that expose scripting, raw video, bracketing, intervalometers, and focus stacking — born exactly because the manufacturers wouldn't

• the **open road**: **Android's Camera2 / CameraX** API (and Google's earlier FCam research camera) expose per-frame control of exposure / focus / gain and access to **raw burst** — which is *why phones, not cameras, became the home of computational photography*

• the **maker road**: a **Raspberry Pi + camera module** (the HQ camera, driven by **libcamera / picamera2**) is a cheap, fully **open and programmable** camera — per-frame exposure/gain control and **raw** access on a hobbyist board — the natural platform for *trying this book's algorithms* and building DIY computational cameras

• **why this matters for the book**: our algorithms (HDR+, burst denoising, computational bokeh) need **programmable capture** — control of the per-frame integral and access to raw; the platform that grants it is where the research happens

• the **histogram** — the distribution of pixel values — is the natural companion to point operations: this short chapter is how to *read* it, and how it can *drive* an automatic tone curve

• a **quick intro** here (full treatment in the HDR chapter): tone mapping = choosing the **mapping from scene tones to display tones**, the natural payoff of point operations

• promoted from a section of [[#Fourier]] into its own chapter (it earns one): a digital image is a **sampling** of a continuous scene, and detail finer than the grid **aliases** — folds down to a wrong, low frequency (moiré, wagon-wheel).

• **aliasing = a high frequency in disguise** (sine sampled below 2×/cycle → looks like a slower sine); 2-D version = moiré on a facade. Figures: `fig-aliasing`, `fig-aliasing-photo`.

• **Nyquist–Shannon**: sample at >2× the highest frequency; the **Nyquist frequency** is half the sampling rate; above it folds down irreversibly.

• **frequency-domain mechanism**: sampling is **multiplication by a Dirac comb** in space, so by the convolution theorem it becomes a **convolution in Fourier** — but by an unusually *simple* spectrum. The comb is **self-dual**: the Fourier transform of a Dirac comb is **another train of Diracs** (spaced at the sampling rate). And that is exactly why sampling is *easy* to reason about in Fourier: convolving by a single Dirac is just a **shift**, so convolving by a *train* of Diracs is just a **sum of shifted copies**. The spectrum is simply **replicated** at every multiple of the sampling rate — no smearing, just shifts. (This is the payoff of Fourier being built on the **shift theorem**: the one convolution that would otherwise be messy collapses into pure translation.) Band-limited copies sit apart; too-wide copies **overlap** = aliasing (`fig-sampling-comb`). Big lesson **L4.13**.

• **cure**: low-pass **pre-filter** (blur) *before* downsampling. Interactive demos: `fig-sampling-1d-demo` (1-D sampling/aliasing/pre-filter, tick-marked samples + spectrum) and `fig-sampling-2d-demo` (downsample the Prudential facade with/without pre-filter → moiré on/off).

• sidebars: aliasing **without** Fourier (pixel = area average); aliasing **on purpose** (KinéCam moiré encodes camera motion).

• **ideal sinc** reconstruction (perfect frequency box ⇒ infinite spatial ripple) is unreachable; practical filters compromise (`fig-sinc-vs-practical`) — handed off to [[#Resampling and upsampling]].

• xref: built on [[#Fourier]]; reconstruction / upsampling lives in [[#Resampling and upsampling]].

• we **often need to compare two images** — fidelity, alignment, an ML training loss

The previous chapter cast recovery as the normal equations $A^\top A\,x = A^\top y$ and made the key point that we **never build $A$** — every solver needs only to *apply* the blur and its transpose. This chapter is the toolkit that actually does the solving: a handful of matrix-free, iterative / multiscale methods, and the rule for which one to reach for. The same machinery returns in [[Poisson image editing]], colorization, matting, and gradient-domain HDR — learn it once here.

💡 **Big lesson (L5.4, as a limit — the data is silent in the null space, and a prior only *invents* there):** an inverse problem can only return what the measurement actually constrained. Where the forward operator $A$ erased information (its null space) or barely kept it (near-null, noise-swamped modes), **no estimator can recover the truth** — the Cramér–Rao bound floors the variance, the data-processing inequality forbids manufacturing information, and the SVD names exactly which modes are lost. A prior (Tikhonov, sparsity, a learned/diffusion model) is the *only* thing that can fill those modes, and what it fills in is **plausible, not measured** — invented from training, not read from this photo. So progress comes in exactly two honest forms: **bring a better prior** (push the wall, accept invention) or **change $A$ at capture** (coded aperture, more exposures, structured light — so the wanted detail is measured in the first place). "AI enhance" is the former wearing the costume of the latter. (Registered in [[Big Lessons]] as the limit-form of **L5.4**; first stated in [[Image priors]] / FUNDAMENTALS, sharpened to an impossibility here.)

reference the ML & deep-learning refresher ([[Refreshers#Machine learning and deep learning]]) up front; this chapter does **not** re-teach networks — it sets up *learning an operator from data* and the **data** that powers it. The concrete deep-network operators are the next chapter, [[Deep learning]].

these are the **deep-network** realizations of the learned-operator framing ([[Machine learning]]); architectures and training are the Refreshers' job — here we survey *what gets learned*.

This chapter is the **generative limit** of the two before it. *Machine learning* replaced a hand-designed prior with a **learned** one (L5.3); *Super-resolution* showed the prior is **not optional** and that a **denoiser is a universal prior** you can plug into any solver (L5.4, PnP/RED, and the punchline that the **score is a denoiser**). Here that same prior becomes something you can **sample from** — a generative model — and the canonical one, **diffusion**, is literally that denoiser run in a loop. We reference the ML / deep-learning refresher ([[Refreshers#Machine learning and deep learning]]) for U-Nets, transformers, and attention; this chapter does **not** re-teach networks — it surveys the *generative* idea and where it plugs back into the book's inverse-problem spine.

• chapter intro: split out of [[Denoising]]; NL-means is the bilateral filter living in patch space, and the reason image self-similarity denoises so well.

• **the shift — filtering → optimization**: instead of one weighted-average *pass*, solve a **global least-squares** problem whose smoothness term uses the affinity as a weight. The same affinity, now inside an energy.

• **colorization** (the demo): the user scribbles a few colors; **propagate** them so that **neighbouring pixels of similar luminance get similar color** — minimize $\sum_p (U_p-\sum_{q\in N(p)} a_{pq} U_q)^2$ over the chrominance $U$, with affinity $a_{pq}$ **large where $I_p\approx I_q$** (small luminance difference) — exactly the bilateral's range idea inside a quadratic energy

• **solve it** with the linear-systems machinery of the regression chapter — sparse, structured, solved by CG / multigrid (cross-ref [[Single-image computational photography#Blind deblurring]], [[Linear Inverse Problems and Regression]])

• the **affinity matrix** here is the same object as the **matting Laplacian** and the graph-cut edge weights → forward-ref [[#Compositing, segmentation and matting]] (matting / segmentation = affinity again)

• also placeable under [[Single-image computational photography#Blind deblurring]] (the "Colorization as optimization?" stub) — keep the *technique* here, cross-ref from there

• **start here — Mertens *exposure fusion*, the gentlest pyramid method**: the cleanest way into this chapter is the technique that does **local tone mapping for free, without ever building an HDR image or choosing a tone curve**. Given a **bracketed stack** (or several renderings of one scene), score each pixel of each frame by simple **quality weights** — **contrast** (Laplacian magnitude, "is it sharp here?"), **saturation** (spread of the RGB channels, "is the color alive?"), and **well-exposedness** (closeness to mid-grey, "is it neither crushed nor blown?") — then keep, per pixel, the best-exposed frame. *(Mertens, Kautz & Van Reeth 2007.)* [`fig-exposure-fusion-mertens`]

• **the catch, and the fix — blend in the Laplacian pyramid**: a *per-pixel* weighted average of the frames **seams and halos** (the weights jump at boundaries between which frame "wins"). The fix is **pyramid blending** ([[Pyramids and wavelets]], Burt–Adelson): build a **Laplacian pyramid of each frame** and a **Gaussian pyramid of each (normalized) weight map**, blend the coefficients **level by level** (each frequency band fused at its own scale, with a correspondingly-blurred weight), then collapse. Because every band is feathered over a scale-appropriate transition, the join is invisible and no halos appear — the same reason pyramid blending hides a seam, now driven by exposure-quality weights.

• **why it opens the chapter**: it is the most intuitive instance of the chapter's one idea — *do the edge-aware / local-tone work **band-by-band in a Laplacian pyramid**, not pixel-by-pixel* — and it needs no range weight, no base/detail split, no radiance map. (Full HDR-capture context and where exposure fusion sits among merge-then-tonemap pipelines: [[Multiple exposure imaging]]; here it is the on-ramp to the pyramid machinery.)

• **from fusing a stack to reshaping one image — local Laplacian filters**: exposure fusion *blends several images* in the pyramid; **local Laplacian filters** (Paris et al. 2011) take the same "manipulate the Laplacian pyramid, halo-free" idea and apply it to a **single** image, reshaping its own bands by a pointwise remapping. Same home (the Laplacian pyramid), one input instead of a stack.

• **the odd one out — not an affinity method**: every other technique here weights *pairs* of pixels by similarity; local Laplacian instead works **band-by-band in a Laplacian pyramid** with a **pointwise remapping** — no range weight, no base/detail split (so none of the halos / gradient reversals those can produce). The multiscale cousin of the bilateral, by a different mechanism. Cross-ref [[Pyramids and wavelets]] (the pyramid it's built on) and [[Bilateral filtering]].

• **the mechanism**: to compute the output's Laplacian coefficient at a pixel and level, take the pixel's value $g$, apply a **remapping function** $r_g(\cdot)$ to the *whole* image (a smooth curve, centred on $g$, that decides what is "detail" vs "edge"), build a Laplacian pyramid of that remapped image, and read off **just this level's coefficient at this pixel**; collapse the resulting pyramid for the result.

• **the knob is the remapping $r$**: near $g$ (small amplitude = **detail**) it **compresses or amplifies** — controlling detail / local contrast; far from $g$ (large amplitude = **edge**) it stays the **identity** — so strong edges pass through untouched. One curve → detail enhancement, tone mapping, or inverse tone mapping, all **artifact-free**.

• **why it matters**: state-of-the-art **edge-aware detail manipulation and HDR tone mapping** with no halos and no gradient reversal — the failure modes that motivated the bilateral in the first place.

• **cost & the fast version**: the naive form rebuilds a pyramid per pixel-and-level (expensive); **Aubry et al. 2014** gives a fast approximation and shows the **unnormalized bilateral** is its single-scale relative (ties back to [[Bilateral filtering#Bilateral filter|the unnormalized bilateral]]).

• **the idea**: like the joint bilateral (filter $I$, take structure from a **guide** $J$) but the output is a **local linear transform of the guide**, $I^{\mathrm{out}}=a_k J + b_k$ fit per window by **least squares** — so it preserves the guide's edges by construction

• **why prefer it**: **$O(N)$**, **no range-quantization artefacts** (the gradient reversal the bilateral can show), and **differentiable** → drops into learning pipelines

• **uses**: edge-preserving smoothing; detail enhancement; **flash / no-flash**; **matting / feathering**; **dehazing** (guide = the hazy image — xref He, Sun & Tang 2009 dark-channel, [[#Dehazing as a prior-driven inverse problem]]); depth upsampling

• the affinity is now **implicit** in the local linear fit — same family, different mechanism; a fitting end-point for the chapter's through-line (bilateral → grid → learned → regression → optimization → guided, all *affinity*)

• **the view**: treat denoising / upsampling as **local regression** — fit a smooth function to nearby pixels, weighted by a kernel

• **steering / adaptive kernels**: shape the kernel to the **local image structure** — estimate the local gradient, then **stretch the kernel *along* edges and squeeze it *across* them** (an anisotropic, data-dependent weight) so it averages along the edge, not across it

• so LARK is the **regression flavour** of the same affinity idea: the kernel geometry is *derived from* local structure, generalizing the bilateral's scalar range weight to an oriented one

• also a **descriptor**: LARK signatures are used for detection / matching (Seo & Milanfar) — note in passing, not developed

• **the third leg of EDGES MATTER** (the part's framing payoff): Poisson **reconstructs from edges**, bilateral **smooths except across edges**, seam optimization **cuts along edges**. State it as the dual of edge preservation: edge-preserving filtering works *hard to avoid* edges; seam methods *go looking for them* — a transition is invisible exactly where the image is already changing, so we hide the cut **in the edges / busy texture** and keep it **out of smooth regions** where a seam would scream. (Callback to the part intro's "respect the edges.")

• the **unifying machinery**: all three sub-families are **discrete optimization on a graph or grid** — a 1-D least-cost path (**DP**), a 2-D globally-optimal boundary (**graph cut / min-cut**), or a spectral partition (**normalized cuts**). What changes between methods is **the energy / affinity**, not the optimizer.

💡 **Big lesson:** *many image edits are **discrete optimization on a graph** — dynamic programming, min-cut/max-flow, or spectral relaxation — and the **energy (affinity) you choose is the whole game**.* The optimizer is off-the-shelf; **what** you penalize (gradient magnitude, label disagreement across edges, region/boundary cost) is the design. This is the **L4.21 affinity** principle (Bilateral) turned into a *cut*: there the affinity said "average these together"; here it says "**don't cut here**." *(Register as new lesson **L6.4 · Image edits as discrete optimization on a graph (the energy is the design)**, first appearance here; "also relevant": Poisson/photomontage, compositing/segmentation/matting, texture synthesis, stereo/MRF labeling. Full box at this section's head; one-line callback to L4.)*

💡 **Big lesson (L16 · Prefilter before you downsample):** a warp that *shrinks* the image (minifies) is throwing samples away — and if you just decimate (keep every $N$th source pixel, or sample a narrow reconstruction kernel) the detail too fine for the new grid **folds down into bold false moiré**, not into nothing. The fix is to **area-average first**: widen the resampling kernel in proportion to the downscale factor $s$ (so it gathers over the whole footprint each output pixel now covers) and renormalize — that low-pass blur is the prefilter. Enlarging needs *no* prefilter (clamp the kernel width to 1); shrinking *always* does. (→ see Big lesson **L16**, first placed in BASIC → Resampling; the frequency-domain *why* is **L4.8** — aliasing folds frequencies above the new Nyquist. The same lesson governs every geometric resample in this part: rectification corners that shrink, stabilization crops, optical-flow rendering.)

⚠️ **Candidate new part-spine lesson (flag only — not registered).** Almost every algorithm in this part follows one shape: **establish a correspondence / coordinate map between two images (or an image and a target frame), then transport pixels along it.** Warping makes the map explicit and applies it; perspective rectification *solves* the map (a homography) from constraints; panorama stitching *estimates* the map from features; morphing *interpolates between* two maps; optical flow *estimates a dense map* from brightness constancy; stabilization *smooths a map over time*; frame interpolation *renders along* a map. A lesson like **"find the coordinate map, then move the pixels"** would unify the part the way L5.4 unifies the inverse-problem part. Worth registering as a part-spine lesson for MOTION/WARPING — *noted here, deliberately not registered* (registry-first convention: register in [[Big Lessons]] before placing).

💡 **Big lesson (L16, anisotropic):** the prefilter-on-minify discipline (**L16**) is *direction-dependent* the moment the warp stops being a similarity. A general warp compresses the source more along one axis than another, so the right prefilter is not a round blur but an **oriented ellipse** — average a lot along the compressed direction, barely at all across it. Isotropic filtering must then either alias the compressed axis or blur the sharp one; anisotropic filtering (EWA and its approximations) does both jobs at once. Same lesson, one dimension richer.

💡 **Big lesson (recurrence — *align before you blend*):** averaging two images only makes sense once their features sit at the *same place*. A plain cross-dissolve blends colors at fixed coordinates, so any misalignment **double-exposes into a ghost**; the cure is to **warp into correspondence first, then average** — the same move as panorama de-ghosting and multi-frame super-resolution's sub-pixel alignment. Morphing is this lesson taken to its limit: not just align-then-average, but interpolate the *alignment itself* over $t$. (Companion to the *capture-then-decide / align-then-merge* family; the dense automatic aligner is [[Optical flow]].)

💡 **Big lesson (L16 recurrence · prefilter the shrinking corners):** rectification does not resample uniformly — to make the *far* (top) of the façade as wide as the *near* edge, the homography **enlarges** the top corners (upsample → just interpolate) and **compresses** other regions (downsample → must prefilter, or alias). So a faithful rectifier applies the **prefilter-before-downsample** discipline *per region*, with the kernel footprint set by the **local Jacobian** of $H$ (how much each area shrinks). (→ see Big lesson **L16**, [[Warping]] / BASIC Resampling.)

💡 **Big lesson (L4.8 here · diagonalize when you can):** a pure translation is *diagonal* in the Fourier basis — it leaves every magnitude untouched and tilts only the phases by a ramp whose slope **is** the shift. So a brute-force $O(\text{window}^2)$ shift search collapses to a single FFT pair: form the normalized cross-power spectrum, inverse-transform, and read the shift off the location of the resulting delta. Whenever the operator is a shift, a convolution, or a correlation, ask first whether the frequency domain turns the work into a per-frequency (diagonal) operation. *(L4.8, first placed in [[Linearity, Fourier, aliasing and deblurring]]; here it turns a search into a transform.)*

The previous part transported pixels along maps that were *given* or *hand-drawn*. This part *estimates* the maps, and the cheapest, most robust way to start is not to match every pixel but to match a **sparse** handful of points you can find again. Three stages, in order: **detect** distinctive points (the *where*), **describe** each one's neighbourhood (the *what*), and **match** descriptors across images (the *which-goes-with-which*). Detect for **repeatability** (the same physical point fires in both images, despite viewpoint, scale, and lighting), describe for **distinctiveness** (the descriptor is unique enough to match unambiguously and invariant enough to survive the change between images). The raw matches are then filtered — ratio test and RANSAC — in the next section.

💡 **Big lesson (the structure tensor, three jobs):** the same $2\times2$ matrix $M=\sum I\nabla I\nabla I^{\top}$ answers three questions across this part. *Is this a good point to detect?* — yes where $M$ has two large eigenvalues (a corner): **Harris / Shi–Tomasi**, here. *Is this patch's motion solve well-conditioned?* — same matrix invertible: **KLT** ([[Feature tracking]]). *What is the local flow constraint?* — $M$ is the normal-equation matrix $A^\top A$: **Lucas–Kanade** ([[Optical flow]]). Detect with it, track with it, solve flow with it — one matrix, read three ways. Corners are special precisely because the *same* both-directions-vary condition that localizes a point also makes its motion recoverable.

The previous section estimated a **dense** flow field — a motion vector at *every* pixel — between *one* pair of frames. This section does the complementary thing: follow a **sparse** handful of *distinctive* points across *many* frames. It is the same physics (brightness constancy, the aperture problem) restricted to the places where the math is well-behaved, and run forward in time. The payoff of the restriction is the chapter's spine: **the matrix that says a patch is a good corner to *detect* is the same matrix that must be invertible to *track* it.** Detect with it (Harris), track with it (KLT), refuse to track where it is singular (Shi–Tomasi). One $2\times2$ matrix, three jobs.

💡 **Big lesson (recurrence of L5.3 — a learned operator swaps a hand-designed prior for one learned from data):** classical KLT tracks a hand-built brightness patch by a hand-derived least-squares update; modern point trackers (PIPs, CoTracker) keep the *exact same task* — "where did this point go?" — but replace the patch with **learned features** and the greedy per-point update with a **learned, occlusion-aware, track-together** module. The skeleton (initialise a trajectory, iteratively refine it against appearance, predict visibility) is unchanged; only the operator is now fit to data. (→ see Big lesson **L5.3**; first appears in [[Deep learning]]. The same "neuralise the classical iteration" move powers RAFT in [[Optical flow]].)

A keypoint matcher hands you a pile of correspondences that is **mostly noise**: repeated texture, occlusion, and ordinary descriptor collisions all produce confident-but-wrong matches, and a single wrong match wrecks a least-squares fit (one bad point pulls the whole model). Two classic, complementary filters make matching usable — one prunes *before* fitting, the other fits *despite* the survivors' remaining outliers.

The classical pipeline — hand-designed detector, descriptor, nearest-neighbour matcher — is exactly the kind of three-stage hand-built system deep learning likes to swallow whole (the **L5.3** move: learn the operator from data). It arrives in three waves. First, **learned local features** that play SIFT's role — joint detector-and-descriptor networks (**LIFT**, **SuperPoint**, **D2-Net**, **R2D2**) trained for repeatability and matchability, often **self-supervised** by warping an image with known homographies and demanding consistent keypoints. Second, **learned matching** that reasons about the *whole set* of features at once instead of independent nearest-neighbours: **SuperGlue** (and the faster **LightGlue**) cast matching as graph/attention-based assignment (an optimal-transport-flavoured solve), while **detector-free** matchers like **LoFTR** skip keypoints entirely and match dense coarse-to-fine. Third, the **3-D-aware** turn — **DUSt3R / MASt3R** regress dense **pointmaps** directly from an image pair, so correspondence, camera pose, and depth fall out together with no explicit keypoint stage at all; matching stops being a separate step and becomes a by-product of predicting geometry. Each wave keeps the *task* and learns more of the *pipeline*.

Once you have descriptors, the cost is dominated not by computing them but by the **nearest-neighbour search** in a high-dimensional space — and exact NN is hopeless at scale (the curse of dimensionality defeats kd-trees past a few dozen dimensions). So the toolkit is **approximate** NN and clever propagation. For sparse descriptors: randomized **kd-trees / FLANN**, **locality-sensitive hashing** (collide nearby points into the same bucket), and **product quantization** for billion-scale retrieval (the lineage that became FAISS / ScaNN vector databases). For *dense* patch correspondence, **PatchMatch** abandons search trees entirely — random initialization plus **propagation** (good matches are spatially coherent, so a neighbour's offset is a great guess) plus occasional random refinement converges astonishingly fast ([[Patch match]]). Worth singling out is **Andrew Adams's random-projection lineage** — Gaussian kd-trees and the **permutohedral lattice** — which finds good high-dimensional matches by projecting onto a few random directions and splatting/propagating on a lattice; the same machinery accelerates the bilateral filter and NL-means denoising ([[Denoising basics]]). The recurring lesson: at scale, *good-enough-and-fast* beats *optimal-and-slow*, and **propagation** (exploit coherence) often beats **search** (treat each query independently).

💡 **Big lesson (the chapter's core — *brightness constancy + the aperture problem*):** track a point by the one thing that's (assumed) conserved — its **brightness**. Linearizing that single scalar equation gives **one constraint per pixel**, $I_x u + I_y v + I_t = 0$, for **two** unknowns $(u,v)$ — so locally motion is **fundamentally underdetermined**: you can only see the component **along the image gradient** (normal flow), never the component **along an edge** (the *aperture problem*). Every flow method is a strategy for supplying the missing second equation — from a **neighbor** (Lucas–Kanade: pixels in a patch share a motion → the well-/ill-posedness is the **same corner/edge/flat structure tensor as Harris**), from a **smoothness prior** (Horn–Schunck), or from **learned context** (RAFT). *Only the gradient carries motion information, and one gradient is never enough.*

The learned-flow story is best told as a single pattern rather than a parade of networks: **take the classical iteration of [[Optical flow]] and *unroll* it into a differentiable network trained end-to-end**, so the hand-tuned data term, smoothness regularizer, and coarse-to-fine schedule become learned. **FlowNet** first showed a CNN could regress dense flow at all. **PWC-Net** made the lineage explicit by folding the *classical* pipeline — **P**yramid, **W**arping, **C**ost-volume — into a compact, fast network: the same coarse-to-fine warping of the previous section, now with learned features and a learned update. **RAFT** is the design that came to dominate — build a 4-D **all-pairs correlation volume** (every pixel's similarity to every other) and apply a **recurrent update operator** that behaves like an unrolled iterative optimizer refining the field; it is the flow analogue of "neuralize the classical solver" and the reason RAFT-style modules now appear in stereo, scene flow, and tracking. Transformer flows (**FlowFormer**, **GMFlow**) reframe the same problem as **global matching**, replacing the local cost-volume update with attention. Throughout, the *skeleton* — features, a cost/correlation volume, an iterative update, coarse-to-fine — is the classical one; learning supplies the operators.

💡 **Big lesson (L5.4 · The prior is not optional):** when the measurement genuinely destroys information — super-resolution past the sensor's sampling, deblurring at killed frequencies, inpainting a hole — *no amount of cleverness recovers it from the data alone*. A **prior** (what natural images look like) is what selects an answer; it is a load-bearing part of the algorithm, not a tuning knob. The honest split: **reconstruction** priors fuse genuinely-measured extra samples; **hallucination** priors *invent* plausible detail that was never measured. (Registered in [[Big Lessons]] as **L5.4**; first appears here; recurs in [[Image priors]] — the denoiser-as-prior generalization — in [[#Blind deblurring]] — blind deblurring & dark-channel prior — and forward in [[Generative AI and diffusion]].)

The previous chapter inverted a **known** blur on a **clean** image — a textbook least-squares solve. Reality breaks both assumptions: sensors add **noise**, and you usually **don't know the blur kernel**. This chapter is what you do then. The unifying frame is one line — *recovery = data-fit + prior* — and each section just swaps in the prior that suits the degradation. Forward-ref the punchline early: when the prior is *learned* rather than hand-designed, this is exactly [[Machine learning]] and [[Super-resolution]] (PnP/RED).

💡 **Big lesson:** *diagonalize when you can* — shift-invariant blur is a **convolution**, so it's **diagonal in Fourier**: deblurring becomes a per-frequency **division** by the kernel's frequency response, and the Wiener filter is just that division, *regularized* per frequency. (→ see Big lesson L4.8, first placed in [[Linearity, Fourier, Aliasing and deblurring]] — recurs here as the reason both inverse and Wiener filters have one-line Fourier forms.)

Haze isn't a deblurring problem — light is **scattered**, not convolved — but it's the same *idea* as the previous chapter in a new costume: an image-formation model that is **under-determined**, made solvable by **one well-chosen statistical prior**.

This chapter is the most literal instance of 💡 L10. A hole has **no measurement at all** — the only thing that can fill it is a model of what images look like. So inpainting is best read as a **ladder of priors**, from the weakest ("be smooth") to the strongest (a learned generative model), and the whole chapter is *which prior, and where it copies-vs-invents*. The recurring honesty: filled pixels are **plausible, never measured** — verifiable when the prior copies genuine structure from elsewhere, an outright **guess** when it hallucinates.

💡 **Big lesson (recurrence of L5.4 · the prior is not optional):** filling a hole is the purest case — there is *zero* data inside it, so **100% of the answer comes from the prior.** Smoothness, self-similarity, a photo database, a learned net — each is a different prior, and the result is exactly as good (and as honest) as the prior is. (→ see Big lesson **L5.4**, first placed in [[#Super-resolution]]; here taken to its extreme — no data, all prior.)