💡 Big lessons

Light, and the physical color it carries, is described by a spectral distribution: the power it carries at each wavelength, a function $S(\lambda)$. One wavelength is a single pure color; real light is a weighted mix of all of them. Everything downstream (how a prism fans light out, how a surface reflects it, how a sensor or an eye turns it into a perceived color) starts from this per-wavelength power.

What we call brightness or intensity is the power of the electromagnetic field, a quadratic quantity of the field. We never record the field itself, only its time-averaged square, so brightness is always non-negative and blind to the field's sign and phase. And because it is quadratic rather than linear in the field, two light contributions do not simply add in brightness: squaring their summed fields leaves a cross term, and whether that term survives or averages away is what coherence decides.

Coherence decides which quantity of light is linear. Coherent light superposes amplitudes, so its cross terms survive as interference; incoherent light superposes intensities, because the random phases average the cross terms away. No source is fully one or the other, only more or less coherent at a given scale of time and space. Ray optics is what emerges in the incoherent, short-wavelength corner of this space: interference washes out, intensities add along paths, and light travels as independent rays of energy. Most photography involves incoherent light, which means that it is linear in intensity.

At a boundary between two media, light bends by $n_1\sin\theta_1 = n_2\sin\theta_2$ — toward the normal entering a denser medium, away from it on the way out. This single rule is the seed of all lens optics: every focal length, every ray trace, the thin-lens equation, and every aberration is Snell's law applied at curved glass surfaces. Pair it with the fact that the index $n$ depends on wavelength (the next lesson) and you also get dispersion. Almost everything optical in this book traces back to it.

The refractive index is not a constant: $n(\lambda)$ depends on wavelength — in ordinary glass it rises toward the blue, so blue bends more than red. Because of this you cannot route every wavelength the same way with a single refraction: a prism fans white light into a spectrum, a raindrop makes a rainbow, and a simple lens focuses blue and red at slightly different distances — chromatic aberration. Much of the craft of lens design (combining crown and flint glass into achromatic and apochromatic groups) exists to fight this one fact. Dispersion is not a defect of cheap glass; it is built into how matter and light interact.

Much of imaging is multiplicative, not additive. The light reaching the eye is illumination times reflectance; contrast is a ratio of intensities; a shadow darkens a region by a factor, not by subtracting a constant. Because the structure is multiplicative, the natural tools are multiply, divide, and log — not just add. This is why, again and again, we will work with ratios and with logarithmic encodings rather than raw linear values. It is also why white balance is hard: to recover an object's true color we must divide out the unknown illuminant $S(\lambda)$ from the product $C(\lambda)$ — undoing a multiplication we never observed directly. (The additive counterpoint — light from different sources adding — and what it means for choosing an encoding, comes in Color technology.)

A diffuse (Lambertian) surface looks the same brightness and color from every viewing direction — its appearance does not depend on where you stand, only on where the light is. This is an idealization (real surfaces add a view-dependent specular sheen), but it is one a huge amount of computer vision quietly assumes, because it lets a scene point have a single color that can be matched across views. Stereo, multi-view and photometric stereo, optical flow's brightness-constancy, and any "just average the color at this point" reasoning all lean on Lambertian reflectance — and shiny, view-dependent highlights are exactly where they break.

The light a surface receives per unit area is thinned by the cosine of the incidence angle: a beam striking at angle $\theta$ from the normal spreads its power over a footprint $1/\cos\theta$ larger, so irradiance $\propto \cos\theta$ (Lambert's cosine law). This $\cos\theta$ is not a fudge factor — it is projected area — and it reappears everywhere: it is why a matte sphere fades toward its edge ($\mathbf n\!\cdot\!\mathbf l$ shading), why the poles are colder than the equator, why lenses vignette toward the corners ($\cos^4$), and it sits inside every radiometric integral over the hemisphere.

Counter to ray-optics intuition, a smaller aperture spreads light more: the diffraction blur (the Airy disk) grows as $\sim\!\lambda N$ with f-number, so stopping down past a point softens the image instead of sharpening it. The flip side is the rule behind every instrument that must resolve fine detail — only a large aperture can diffract little and resolve finely — which is exactly why microscopes and telescopes are built around large lenses and mirrors, and why a phone camera's tiny aperture hits a hard resolution wall it cannot pass. It is the optical cousin of "you can't sample your way past Nyquist."

Forming an image is, at bottom, controlling which ray (or rays) of light contribute to each point of the image. A bare sensor fails because every scene point sends light to every pixel. The sum is a flat gray. A pinhole admits one ray per point (sharp, but dim); a lens gathers a whole cone of rays from one scene point and bends them back to one image point (bright, but only for one focal plane); an open hole with no focusing mixes many rays and blurs. Every imaging idea in this book (focus and depth of field, coded apertures, light fields, even a CT scanner) is a different answer to the one question: which rays land here?

For a pinhole a fixed distance $f$ from the sensor, the field of view is set by the sensor size: $\text{FOV}=2\arctan(\text{sensor}/2f)$. Hold $f$ fixed and slide a larger sensor behind the same pinhole and you take in more of the world; a smaller sensor crops to a narrower view. So "how wide a shot is" is never a property of the pinhole (or, next chapter, the lens) alone: it is the pinhole-to-sensor distance and the sensor size, together. This is the geometric root of the crop factor and of "35 mm-equivalent" focal lengths (next chapter), and the reason the same lens is wide on a big sensor and tele on a small one.

From a pure geometric-projection standpoint, cropping an image and zooming in (lengthening the focal length) are one and the same operation. Both leave the perspective projection (the divide-by-$Z$ onto the normalized plane) completely untouched and change only the pixel stage $K$: a crop keeps a smaller patch of the projected image and rescales it to fill the frame, which is exactly what a larger $f_x, f_y$ does. So a 2× crop and a 2× zoom give the identical framing and identical perspective, differing only in resolution: the crop simply has fewer pixels to show for it. This is why "digital zoom" is geometrically free but costs detail, and why the crop factor between sensor sizes acts as a focal-length multiplier.

With the camera at the origin looking down $z$, a scene point $(X,Y,Z)$ lands at $(f X/Z,\; f Y/Z)$: the whole of perspective is dividing by depth. That one division produces everything we call perspective: farther things appear smaller (double the depth, halve the size), parallel lines converge to vanishing points, and the picture gains its sense of space. It is also why depth is the hard thing to recover: the division throws $Z$ away, and a single image cannot tell a small near object from a large far one. Every stereo, structure-from-motion, and depth-estimation method in later parts is, in effect, trying to undo this divide.

A camera in any position and orientation projects a 3D point in two moves: a rigid transform that brings the world into the camera's frame (because moving the camera equals moving the world by the inverse), followed by the perspective divide. In homogeneous coordinates the two compose into a single $3\times4$ matrix $P$, applied to the homogeneous point $(X,Y,Z,1)$, and the pixel is recovered by dividing by the last coordinate. So any pinhole view, however the camera is placed, is "one matrix multiply, then one division." That divide is the only nonlinear step in all of projection, the generalization of L2.13's divide-by-depth to an arbitrary viewpoint.

You can make a subject fill the same fraction of the frame either by zooming in or by stepping closer, and at that one depth the two give the identical magnification. But magnification falls off with distance, so everything at other depths behaves differently. Changing focal length rescales the whole projected image by a single factor (it is the pixel stage $K$, the "crop = zoom" lesson again), preserving every near-far size ratio; changing your distance rescales near and far by different factors, because each object's magnification depends on its own depth. Step close and the near looms while the far recedes; zoom from afar and the background stacks up "compressed" behind the subject. So relative near-far size (what we call perspective or compression) is set by viewpoint (distance), not by the lens; the focal length only chooses how much of that view to crop. The dolly zoom below is this lesson made vivid.

The field-of-view-depends-on-sensor-size lesson (L2.11) carries straight over to a real lens: for a fixed focal length $f$, the field of view still follows $\text{FOV}=2\arctan(\text{sensor}/2f)$, so the same lens is wide-angle on a big sensor and telephoto on a small one: the smaller sensor simply crops the projected image into a narrower view. A focal length in millimeters therefore says nothing about framing until you also name the sensor, which is why focal lengths are quoted as a 35 mm- (full-frame-) equivalent: the focal length that would frame the same way on a $24\times36$ mm sensor. Concretely, a phone's main camera has an actual focal length of only about 6 mm, but on its tiny (~1/1.3-inch) sensor it frames like a 24–26 mm lens on full frame, so it is sold as "26 mm equivalent." The conversion is just the crop factor, full-frame diagonal ÷ sensor diagonal (≈ 7× for that phone), multiplying the real focal length.

Where a pinhole keeps only one ray, a lens gathers a whole cone of rays leaving a scene point and bends them back to a single image point, but only for points at the focus distance. That is the lens's compromise: vastly more light (a bright image) in exchange for a single plane of sharp focus, with everything nearer or farther spread into a blur disk (the circle of confusion). The aperture sets how wide that cone is, and hence the depth of field. Every later idea about focus, bokeh, refocusing, and light fields is a variation on "which cone of rays reconverges where."

What we just did to Snell's law is one of the most powerful moves in all of physics and engineering, and it is worth naming as a general tool, because the book will reach for it again and again. Replace a messy nonlinear function by its tangent line near the operating point, keep only the first-order (linear) term of its Taylor expansion, and it is an excellent approximation as long as the relevant quantity stays small. Here the small quantity is the angle, $\sin\theta\approx\theta$, and the reward is the entire clean, linear machinery of paraxial optics. The same reflex linearizes brightness change into the optical-flow constraint (a chapter away), drives the Gauss–Newton steps of lens optimization and bundle adjustment, and propagates small noise through a pipeline. Two disciplines keep it sound. Know when it breaks: the approximation is only as good as the terms you dropped are negligible, so always ask "how small is small?" And keep the next term in your pocket: restoring it is not a footnote but a whole theory: for optics that next term is the cubic $-\theta^3/6$, and restoring it is the study of aberrations (next big lesson, and the OPTICS part).

The clean thin-lens equation $1/f = 1/u + 1/v$ is not a separate law: it is Snell's law L2.4 linearized, kept to first order under the small-angle approximation $\sin\theta\approx\theta$, the regime known as paraxial (rays near the axis, gentle angles, one wavelength). Because it keeps only the linear term, it tells you exactly when the idealization holds and exactly how it breaks: restore the next term $\sin\theta\approx\theta-\theta^3/6$ and the Seidel aberrations appear; let $n$ vary with wavelength and chromatic aberration appears. "Thin lens," "first-order optics," "a linear (paraxial) lens," and "a lens with no aberrations" are all the same approximation, and the OPTICS part is the story of paying back what it borrowed.

Parallel rays through a thin lens converge to a point at the focal length, and rays going through the center of a thin lens do not deviate. The first rule is what focal length means (where a far subject's parallel bundle meets) and carries the lens's focusing power; the second carries its perspective, identical to the pinhole's. Between these two construction rules and similar triangles, you can derive all of thin-lens optics.

The thin-lens equation $\tfrac{1}{u}+\tfrac{1}{v}=\tfrac{1}{f}$ is symmetric in $u$ and $v$: swap them and it is unchanged, so object and image are interchangeable. Send the light back the other way and the rays retrace exactly the same path (the reversibility of light, made arithmetic). Two consequences fall straight out: an object at infinity images at $v=f$, which is what focal length means, and focusing a camera is nothing but moving $v$ until the chosen object's conjugate lands on the sensor.

The most basic way to focus follows straight from the equation: move the whole lens relative to the sensor, changing the lens-to-sensor distance $v$ until the chosen subject's conjugate lands on the sensor. This is full or unit focusing, the entire optical unit sliding with its focal length fixed. A bellows, the helicoid focus ring on a simple prime, and a bare enlarger lens all work this way, and it is the mental model to keep. It is only the simplest option, though. A compound lens has many elements, and moving just some of them (front-cell, internal, or rear focusing, sometimes with floating groups) can focus while extending the barrel less, holding aberrations in check across the focus range, or even nudging the effective focal length. Those techniques, and why designers reach for them, come in the OPTICS part's focus section.

Magnification is the size of a subject's image divided by the subject's actual size in world units (meters): $m = -v/u$ is exactly that ratio (image height over object height). Photographers quote it as a reproduction ratio: 1:1 ("life size") means the image on the sensor is as large as the subject itself; 1:2 means half life size. Crucially, what fraction of the frame a subject then occupies depends on the sensor size: the same magnification fills more of a small sensor than a large one. On a full-frame ($24\times36$ mm) camera a $1{:}2$ reproduction images a subject about $48\times72$ mm to exactly fill the frame; put that same subject on a smaller sensor at the same magnification and it overflows. So magnification is a property of the optics-and-distance alone, while "how big it looks in the picture" also needs the sensor size.

$N = f/D$: the f-number tells you the width of the opening measured in focal lengths, not in millimeters. $f/4$ means the opening is one quarter of the focal length across. Writing the aperture this way is what makes it a portable brightness setting: because the cone of light converging on each sensor point has half-angle $\tan\theta = 1/(2N)$, the ratio alone sets the image brightness, so the same f-number gives the same exposure on any lens, whatever its focal length or physical size. The denominator is why a small number is a big opening, and the area-goes-as-diameter-squared law ($\text{light}\propto 1/N^2$) is why the stop series climbs by $\sqrt2$.

No pixel measures "the" value at a point in space, time, or color — it reports an integral over everything the photosite cannot tell apart. Four axes appear explicitly above: the spatial footprint (the pixel's area, plus the optical blur smeared across it), the exposure time (a window, not an instant), the spectrum (wavelength, weighted by the channel's spectral sensitivity), and the solid angle of rays the aperture admits; a fifth, polarization, is usually lumped in too. Naming all the axes at once is what makes motion blur, defocus, color, vignetting, and aliasing the same phenomenon — a finite integral standing in for an idealized point sample — and the width of the integral along each axis is a knob the rest of the book spends its time controlling.

A defocused point never images as a point but as a blur disk: the circle of confusion. The depth of field is then defined as the range of distances over which that disk stays smaller than a chosen maximum $c$: inside the depth of field every point is blurred less than $c$, outside it more. The operative word is acceptable: $c$ is not a constant of nature but a judgement call that depends on the target application, the viewing distance, and the sensor (or print) resolution. A billboard read from across the street tolerates a huge $c$; a 100% crop on a 4K monitor almost none. Change the standard and the depth of field changes with it, even though the optics did not.

Away from the macro range (subject much closer than the hyperfocal distance), the total depth of field follows a clean scaling, $\text{DoF}\approx \dfrac{2\,N\,c\,s^2}{f^2}$: it grows linearly with the f-number $N$ (stop down for more), quadratically with the focusing distance $s$ (step back and it balloons), and inversely with the square of the focal length $f$ (wider for more, longer for less). These three exponents explain at a glance why a phone (tiny $f$) keeps almost everything sharp, why close focus annihilates depth of field, and why "I need more in focus" always reduces to stop down, back up, or go wider.

The convenient first-order results of this chapter (the depth-of-field scaling just above, the framing shortcut "magnification $\approx f/\text{distance}$", even the bare f-number as a stand-in for image brightness) all quietly assume the subject is far compared with the focal length. Up close (macro, at magnifications near or past 1:1) they break. The pupil magnification stops being 1, the effective f-number grows as $N(1+m)$ so you lose real light, the working distance shrinks toward the front element, and the depth of field collapses to millimeters. In the macro regime you must drop the shortcuts and return to the exact conjugate geometry (and, often, to focus stacking). It is a good reflex to ask of any optics approximation: does this still hold when the subject is this close?

There is a single focus setting that extracts the maximum depth of field from a lens: the hyperfocal distance $H = f^2/(N c)$. Focus there and the far limit runs all the way to infinity while the near limit sits at exactly $H/2$, so the entire scene from $H/2$ to the horizon falls inside the acceptable circle of confusion. Focusing at infinity instead throws away the whole sharp zone between $H/2$ and $H$ for nothing. It is the landscape-and-street photographer's standard move for front-to-back sharpness, and the basis of zone focusing (pre-set the focus, shoot without refocusing).

Hold the framing and the f-number fixed and scale the camera down: the defocus blur of the background, as a fraction of the frame, shrinks linearly with the sensor size, because it tracks the lens's physical aperture diameter $D=f/N$ and $f$ shrinks with the sensor. So shallow depth of field is a privilege of physically large apertures (a big sensor with a big lens) and a small sensor cannot fake it optically. This is why phones have deep depth of field and must compute fake background blur (portrait mode, a later chapter): the tiny sensor shrinks every blur disk toward nothing. It is the image-space twin of "small sensor → more depth of field" above.

Camera rotation (yaw/pitch) smears the image by $b_{\text{px}} \approx f_{\text{px}}\,\theta$, the same for every object regardless of depth. Camera translation and subject motion smear it by $b_{\text{px}} \approx f_{\text{px}}\,(\Delta \text{ or } v\Delta t)/D$, which shrinks with distance. So for ordinary distances rotation dominates, and only up close (macro) does translation take over. The focal-length dependence divides the same way: yaw and pitch blur scale with $f_{\text{px}}$, so a longer lens (higher magnification) turns the same angular wobble into proportionally more blur (the basis of the $1/f$ hand-holding rule), while roll blur $b \approx r\,\theta_{\text{roll}}$ is fixed by image position $r$, not by focal length. Magnification amplifies yaw and pitch but never roll, which is why yaw and pitch usually dominate the shake blur, the more so the longer the lens. And because roll and perspective make $b$ vary across the frame, camera-shake blur is spatially non-uniform: a single blur kernel cannot describe or undo it.

A photosite is at heart a photon counter: each absorbed photon frees a charge carrier — an electron, or equivalently a hole (a missing electron) — and the well accumulates them in direct proportion to the light that arrives. So the raw signal is linear in scene radiance, before any gamma or tone curve. This linearity is not a detail; it is what makes raw arithmetic trustworthy: HDR merging adds exposures, white balance is a per-channel multiply, flat-field correction is a divide, and all three are valid only because the numbers are proportional to light. Display encoding (gamma) deliberately hides this linearity for the eye — so computation should undo it and work in linear light.

Almost every camera trade-off scales with the sensor's linear size, captured in the crop factor $c$ (full-frame diagonal $\div$ the sensor's). Three consequences, each with a different scaling: - Field of view — to match a framing, the equivalent focal length scales linearly, $f_{\text{eq}} = f \times c$ (L2.11, L2.16): a phone's $6$ mm lens frames like $\sim\!26$ mm. - Depth of field — at fixed framing and f-number the physical aperture diameter, and so the background blur, scale linearly with sensor size, so the equivalent f-number is $N_{\text{eq}} = N \times c$ L2.30: a phone at $f/1.8$ has full-frame's $f/12$ depth of field. Small sensors give deep, near-universal focus and little bokeh. - Light and noise — at equal framing and f-number the total light collected scales with sensor area ($\propto c^{2}$), so the photon count does too; photon-noise-limited SNR and dynamic range then scale with the linear dimension ($\sqrt{\text{area}}$, $\propto 1/c$) L2.35. This total-light advantage, not pixel count, is why a big sensor looks cleaner. So the bookkeeping is: equivalent focal length and equivalent f-number are linear in sensor size; total light gathered is quadratic (its area); the SNR and dynamic-range benefit go as the linear dimension (the square root of area). One dial, three different exponents.

Two numbers describe noise spread, and it pays to know why both exist. Variance (the mean squared deviation) is the quantity the mathematics wants: it is additive for independent sources — exactly what we just used, and the basis of the $1/\sqrt N$ averaging law in Denoising basics — it is the second moment that falls out of every Gaussian and least-squares derivation, and it composes cleanly under linear operations. But its units are squared (squared gray-levels, squared photons), so it means nothing to the eye. The standard deviation $\sigma$ (its square root) is the one people should read: it is back in the original units, directly comparable to the signal — the "$\sigma$" of error bars and the grain you actually see. Rule of thumb: derive and combine in variance, report in standard deviation.

Even a flawless sensor is noisy, because light itself is grainy: photons arrive as discrete, randomly-timed events, so the count in any fixed interval follows Poisson statistics ($\sigma=\sqrt N$), and no electronics can remove it — the randomness is in the light, not the chip. The only true cure is collecting more photons: a bigger pixel, a wider aperture, a longer exposure, or averaging many frames. Crucially, raising ISO is not a cure — it is gain applied after capture, amplifying signal and shot noise together. This irreducible floor is why bigger sensors win, why shadows are noisy, and what every denoiser later in the book is ultimately fighting.

Sensor noise is a sum of distinct sources whose variances add, but each rules a different regime — so diagnosing a noisy image means asking which one you are seeing. Photon shot noise ($\propto\sqrt{\text{signal}}$) dominates the midtones and highlights, and any well-lit shot. Read noise is signal-independent and dominates the deep shadows and short exposures, setting the floor. Thermal / dark-current noise grows with exposure time and temperature — the long-exposure and astrophotography enemy. Quantization matters only with too few bits L2.44. And fixed-pattern noise / PRNU — static per-pixel gain and offset variation — reads as a structured pattern and is removed by calibration (dark and flat fields). Match the symptom to the source before reaching for a fix.

Add the readings of $k$ small pixels and you get the same Poisson (shot) noise as one big super-pixel that caught the same photons — but you pay the read noise $k$ times, once per read-out, instead of once. So the only fundamental advantage of a bigger pixel (or one longer exposure) over many smaller ones is the read noise, not the photon statistics — as long as no light is lost in the gaps. The same holds for aggregating along any dimension: summing $k$ short exposures, or $k$ burst frames, keeps the shot noise of the total light but charges read noise $k$ times. This is precisely why charge-domain binning (combine, then read once) beats digital summation (read $k$ times, then add) in the dark.

Shot noise is Poisson, so its variance grows linearly with the signal ($\sigma^2 \propto N$) and its standard deviation grows only as the square root ($\sigma \propto \sqrt{N}$). Two consequences pull in opposite directions: the absolute noise is larger in highlights than in shadows (a 10,000-photon highlight has $\sigma=100$; a 100-photon shadow has $\sigma=10$), yet the SNR $= N/\sqrt{N} = \sqrt{N}$ is higher in the highlights — the signal outgrows the noise. So bright areas are noisier in absolute terms but cleaner to the eye; the shadows have the worst SNR (a figure, a figure). This is why exposing to the right and brighter scenes look smoother, and why we gamma/log-encode so equal ratios get equal steps.

The variances add, and two terms dominate: shot noise contributes an amount proportional to the signal (Poisson, variance = mean, scaled by the gain) and read noise a constant floor. So the total is variance ≈ gain·signal + read² — a straight line in brightness (a figure). The slope is the photon gain (it steepens with ISO, which amplifies the signal and its shot noise together); the intercept is the read-noise floor. We can measure it without any model: take an aligned burst of a static scene and, per pixel, plot the variance across frames against the mean. This affine law is a statement about the variance and is essentially exact for the two dominant sources; it says nothing yet about the shape of the distribution. Where the electron count is large, a single pixel's value over many frames does look ≈ Gaussian (a figure), which licenses modeling per-pixel noise as additive Gaussian in the midtones and highlights. That bell is only an approximation, though: the shot term is really Poisson (variance = mean, but discrete, skewed, and non-negative), and in the deep shadows, where only a handful of electrons land, it stops looking Gaussian at all. The additive-Gaussian model is a convenience of the well-exposed regime, not a law of noise in general (we pin down where it breaks below).

A recorded value is clamped to $[0,\text{max}]$. Near black the negative half of the noise is cut off and frames pile up at 0; near white the positive half is. So at the extremes the noise distribution is asymmetric, with its mean pushed inward (a figure). The consequence for denoising: a naive average or smoothing returns that biased mean, so it makes shadows come out too bright and highlights too dark — a bias every denoiser must correct for (we return to it in Denoising, BASIC).

The top rail is not a hard wall. Read noise is added after the charge saturates at the well (clip 1) and before the ADC clamps (clip 2), so a pixel pinned at full well is digitized to $g\,N_{\text{full}}+B$ plus a zero-mean Gaussian excursion: about half of all saturated pixels read strictly below $\mathrm{DN}_{\max}$, by up to a read-noise's worth. It makes no difference how far over-range the pixel truly is: a highlight ten times too bright still only fills the well to $N_{\text{full}}$, gets the same read noise, and lands below the maximum half the time. So a measured value just under the ceiling does not prove a pixel was unsaturated, and a patch of genuinely clipped highlights reads a little spread of values, not one flat number. Anything that must know what saturated (HDR-merge pixel rejection, highlight reconstruction, clipping warnings) has to threshold a few $\sigma_r$ below $\mathrm{DN}_{\max}$, never exactly at it.

A single measured value is built in four stages, in the order the sensor applies them: (1) Poisson photon shot noise, whose variance equals its mean, the irreducible floor; (2) saturation at the well, a hard clip of the electron count at the full-well capacity; (3) additive Gaussian read noise, the electronics' contribution, applied after the charge is held; and (4) the ADC clip to $[0,\mathrm{DN}_{\max}]$, after gain and the black-level pedestal. Do it in electrons, the only unit in which shot noise is exactly Poisson. The tidy affine Gaussian ($\sigma^2=a\,I+b$) is just the well-exposed limit of this recipe with the Poisson shape and both clips dropped, which is why it matches in the midtones and parts from the truth in the deep shadows (skewed, floored) and at saturation (a soft, spread-out ceiling). The order matters: because read noise lands between the two clips, saturated highlights do not pile up as a clean spike (see the two clipping lessons above). Get it right and you reproduce real noise; get it wrong and your simulator, training data, or denoiser is fitting a sensor that does not exist.

A single exposure records from a top — the photosite's full-well capacity, where it saturates and clips to white — down to a bottom — the noise floor, the read noise that drowns the shadows. Their ratio is the dynamic range, quoted in stops. That floor is not absolute: it sits wherever the SNR drops below what you deem acceptable, so, just as with depth of field and its circle of confusion, the number depends on the quality criterion you pick (SNR = 1 gives the generous engineering range; a stricter SNR gives the narrower usable one). You widen it with a bigger well (larger photosites — why a full-frame sensor out-ranges a phone) or a lower floor (cooling, lower read noise), and you beat it altogether by merging multiple exposures (HDR). This is the capture-side companion to the quantization lesson below: what bounds a single shot is this range — full-well over floor — not the number of bits.

With enough bits and a sane encoding, what limits image quality is noise and dynamic range — the two quantities of this section — not the number of levels the analog-to-digital converter (ADC) offers. The noise floor and the full well bound what you can record; the quantization step, once it sits below the noise, is invisible. (Indeed, a little noise dithers the quantization, hiding it further.) The one place levels do bite is banding in the deep shadows of a linearly-encoded image — too few codes where the eye, working in ratios, is most sensitive — and that is exactly the problem gamma encoding exists to solve, by spending code levels perceptually (Measuring and encoding color). So "rarely" is not "never." We meet this lesson again in BASIC → Image representation → Float vs 8-bit, where floats with headroom are the sane default.

Inverting $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ is hard for exactly two reasons: the operator and the noise. The noise $\mathbf{n}$ corrupts every measurement, so even a perfectly invertible $A$ hands back a corrupted answer. And the operator $A$ throws information away: it can be rectangular (fewer measurements than unknowns, as in the Bayer mosaic, which records only one color channel per pixel) or ill-conditioned: technically invertible but crushing some directions of the scene almost to zero, as in the blur from camera shake, so undoing it divides by almost nothing and detonates the noise. Two further twists make it worse: sometimes $A$ is itself unknown and must be estimated alongside the scene (blind deblurring), and even when it is nominally known, modeling $A$ precisely is hard: a subtly wrong operator yields confident artifacts. Everything downstream, the regularization below and the whole inverse-problem chapter that follows, is the fight against these.

Strip away the glass and an imaging system is one linear map: it takes the scene $\mathbf{x}$ — written either as a flat billboard at a fixed depth or, more generally, as ray space (the light field) — and returns the image $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, a linear projection of the scene plus sensor noise. The matrix $A$ is the camera: pinhole, lens, stereo, coded aperture, the motion-invariant sweep — they differ only in their rows. And because $A$ is a design rather than a fact of nature, building an optical system is choosing $A$: the whole of computational optics is engineering the operator — in space, in time, in ray space — together with the reconstruction that inverts it, to pull better images and extra information out of the light. Everything after this chapter is organized by one question about that operator: how invertible is $A$?

Natural images occupy only a tiny fraction of the space of all arrays of RGB values.

If we can computationally model the manifold of natural images, we can use it to exclude unlikely solutions to inverse problems.

Factorization problems seek to decompose a number, or an image, into a product of two unknown pieces (scalars, or whole matrices) given only the product. This is fundamentally ill-posed: many factor pairs multiply to the same result, and there is always the degenerate "put everything in one factor, leave the other the identity" split. Only a prior on each factor breaks the tie. In the fully general case it is our linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, except we must now solve for the operator $A$ as well as the unknown $\mathbf{x}$. Examples include white balance, intrinsic images, matting, and blind deblurring.

Each of the three cone types reports one number: the integral over wavelength of the incoming spectrum times that cone's spectral sensitivity — a projection (dot product) of the spectrum onto the cone curve, $r_k=\sum_i c_k(\lambda_i)\,E(\lambda_i)=\mathbf c_k\!\cdot\mathbf E$. Three cones → three dot products → the entire input to human color. Everything else about color — metamerism, "primary" colors, the color-matching matrices of Measuring and encoding color — follows from this one fact: color is three dot products with three fixed curves.

Because color is only three dot products (L3.1), the map from spectrum to response has a huge null space: infinitely many different spectra collapse to the same triple and look identical — they are metamers. This is at once bad news — the eye throws away almost all spectral information, and two objects that match under one light need not be the same physically — and good news — you can reproduce any color with just three primaries, which is the only reason displays, prints, and the whole of color technology can exist.

A metameric match is a match between two products of reflectance and illuminant (L2.6). Swap the illuminant and you multiply by a different spectral shape, so two surfaces that looked identical under one light can diverge under another — illuminant metamerism. It is the headache behind prints that shift between the proofing booth and the living room, and fabrics that match in the shop but clash in daylight; color-critical work must always name the light a match is judged under.

Human color is not computed in one step. Stage 1 is trichromatic — three cone projections (L3.1). Stage 2 recombines them into opponent channels — roughly luminance ($L+M$), red–green ($L-M$), and blue–yellow ($S-(L+M)$) — that is, differences, not raw cone signals. This two-stage structure is why there is no "reddish-green," why staring at red leaves a green afterimage, why "primary color" is so confused (cones vs. opponent hues), and why luma/chroma encodings and chroma subsampling work — luminance carries the detail, the chroma differences are smooth.

The cone "axes" overlap heavily (L and M are nearly redundant) and light cannot go negative — and those two facts are what make color algebra genuinely awkward. With an orthonormal basis, finding a vector's coordinates is just projecting onto the axes; here it is not. Recovering the coordinates of a non-orthogonal basis requires a so-called dual (reciprocal) basis, and for a positive, overlapping basis the dual vectors point partly into the negative quadrants — the natural analysis directions have negative coordinates, which physical light can never supply. This is the deep reason there is no perfect set of primaries to reproduce all colors. (→ see Big lesson: Human color is non-orthogonal and non-negative.)

CIE XYZ, the LMS cone responses, and every linear RGB working space (sRGB-linear, Adobe RGB, ProPhoto, a camera's native raw space) are linear transforms of one another — a single $3\times3$ matrix (plus, for a real camera, a fitted approximation). Converting between color spaces is therefore mostly a matrix multiply, and the apparent zoo of color spaces is really one 3-D vector space written in different bases. The non-linear perceptual spaces — CIELAB, the gamma curve — are the deliberate exception, a separate step layered on top (the next sections).

Nearly every image file on disk (JPEG, PNG, most web and camera output) stores gamma-encoded values, usually sRGB, not linear light. (The notable exceptions are camera RAW, which is scene-linear, and the log formats of professional video, S-Log / Log-C / V-Log and the like; both are working formats meant to be developed or graded, not shown directly.) The display pipeline assumes exactly this: the monitor, its driver, and the operating system together apply the inverse curve (roughly a $2.2$ power) so that the stored code values become the right amount of linear light at your eye. That is the standing encode → store → decode-at-display contract. It bites the moment you compute in linear light (blending, blurring, tone-mapping, physically-based rendering, or the sensor-noise simulator of the noise chapter): your result is linear, but if you write it straight to the screen the display still applies its decode, darkening it and crushing the shadows. So you must gamma-encode a linear image before displaying it, and symmetrically decode an encoded image to linear before doing linear-light math. Skipping either half is the single most common cause of images that come out mysteriously too dark or washed out.

Whether light adds or multiplies should dictate how you encode it. Light from independent sources adds (two lamps, the blur of an out-of-focus lens, the accumulation of photons on a sensor) — these are linear operations, and they are correct only on linear-light values, which is why deconvolution, resizing, and physically-based blur must decode the gamma first. Surface reflectance and perceived contrast, on the other hand, multiply (a gray card under twice the light, a filter cutting a fraction of each wavelength) — and a log encoding makes the multiplicative native, turning products into sums. Gamma is the pragmatic compromise: a power law that behaves better than $\log$ near zero (where $\log$ blows up) while still matching perception. Get this wrong — average gamma values, or sharpen in log — and you get milky blurs, wrong colors, and crushed shadows. The same additive-vs-multiplicative split organizes tone mapping, HDR, and point operations later (→ Big Lessons).

Strip away the color filter and a sensor is monochrome at the pixel: each photosite reports a single number, the count of photons it caught — not a color. The three (or more) values that color needs must therefore be multiplexed along some other axis: over space (a color-filter array such as Bayer, paying with resolution and a demosaicking step), over time (sequential color filters, paying with motion robustness — the old color-wheel and many scientific cameras), across multiple sensors (a beam-splitter prism feeding three chips, the 3-CCD video camera, paying with bulk and cost), or over depth (wavelength-dependent absorption in stacked photodiodes, the Foveon, paying with noise). Every color camera is a choice of which axis to spend. The same constraint shapes the eye, whose three cone types sample color over space at the retina (L3.1, L3.5).

The visual system's goal is to report the reflectance of surfaces (lightness, and its chromatic twin, color) and to be invariant to the illumination. Because the light reaching the eye is the product illumination × reflectance (L2.6), that invariance means dividing out the multiplicative illuminant and keeping only the surface term. This is not a quirk but an ecological design: survival depends on identifying the material world (predator or shadow, ripe or unripe, stone or water), which is a property of reflectance, while the illuminant is a nuisance variable that changes from dawn to dusk. Lightness constancy, color constancy, and the white balance a camera must perform (next) are all the same act: discount the light to see the object underneath.

The chromatic twin of L3.10: just as the visual system reports lightness and discounts the overall level of the light, it reports surface color and discounts the color of the light. A white shirt reads as white under blue sky, warm tungsten, or green shade, though the spectrum reaching the eye differs enormously. The front-line mechanism is von Kries adaptation — an independent gain on each cone type (L, M, S), the direct biological ancestor of a camera's white balance — backed up by spatial and contextual cues and by memory colors. The discounting is automatic and mostly unconscious, which is exactly why a color cast a camera faithfully records looks wrong to us: our own vision would have removed it before we ever noticed.

Color constancy is an inference from cues, and a photograph both starves those cues and dual-codes the light: a picture is at once a flat object in your room and a window onto another scene under its light. Faced with the conflict, the visual system adapts largely to the display or print and its surround, not to the illuminant inside the picture. So the viewer does not discount the depicted light the way they would standing in the scene: a tungsten-lit photo still looks orange even though a tungsten-lit room would not. That is the mechanistic reason white balance must be performed in the camera or pipeline — the ISP substitutes for a constancy step the viewer, looking at a picture, can no longer run.

The eye's sensitivity to detail depends on spatial frequency: the contrast sensitivity function (CSF) peaks at a few cycles per degree and falls off toward both higher and (gently) lower frequencies — band-pass for luminance. Crucially there is not one CSF but several: the chromatic CSFs (red–green, blue–yellow) cut off at far lower frequency and are nearly low-pass, so we see fine brightness detail but blur fine color detail (the curves of a figure). This single family of curves underwrites chroma subsampling, JPEG's quantization tables, perceptual image metrics, and the rule of thumb: spend your bits on luminance, not chroma.

We resolve fine brightness but only coarse color — and the reason is where the channels are computed, not the cones themselves. Luminance is the dense sum $L+M$, carried by small-field, center–surround pathways → high spatial bandwidth. Chrominance is the opponent differences $L-M$ and $S-(L+M)$, formed by the in-retina/LGN opponent recoding (L3.4) that pools cones over larger areas — and the blue–yellow channel is further limited by the sparse S-cone mosaic. Pooling a difference is low-pass filtering, so the chromatic CSFs of L3.13 cut off far earlier than the luminance one as a direct consequence of the retinal wiring. Practically: spend resolution on luminance, not chroma — chroma subsampling and luma/chroma codecs are engineering the eye's own design back in.

The eye integrates light over time, so its temporal resolution is limited. Above the critical flicker-fusion frequency (around 50 to 60 Hz for us, higher in bright light) a flickering source fuses into a steady glow, and a rapid enough sequence of stills reads as continuous motion (apparent motion and persistence of vision). That single fact makes film, video, and displays possible, sets the frame and refresh rates they need, and defines the sensitivity-versus-motion-blur trade (a longer integration gathers more photons but smears anything that moves) that recurs from the retina to the camera shutter. Like the spatial CSF, the spatiotemporal response has been measured and fit into a computable visibility function, which is what lets encoders and displays spend effort only where the eye can follow.

Much of low-level human vision is not vague or merely qualitative — it has been measured by psychophysics and fit by closed-form, evaluable models: the contrast sensitivity function (Mannos–Sakrison, Barten, Mullen, Kelly, stela/castleCSF), light adaptation, the opponent transform, contrast masking, flicker fusion, and the CIE color-matching machinery. That means perception can be put inside an algorithm — a number that predicts what a viewer will and will not see — which is what lets compression, tone mapping, quality metrics, and display design optimize for the eye instead of for pixels. The threshold of visibility is a computable quantity. (The caveat is real: these are population-average models calibrated under specific conditions — see L3.14 and the human-factors part — and higher-level perception, taste, and meaning are nowhere near this tractable.)

Only the fovea (a patch a degree or two wide) sees in high resolution, so the eye cannot take in a scene at once. It samples through a rapid sequence of fixations aimed by saccades, attending to essentially one location at a time and assembling the whole from a few sharp glimpses spread over time. Where the gaze lands is not random but driven by saliency (faces, motion, contrast, the unexpected) and by the task — exactly what a painter or cinematographer steers through composition, and what saliency and gaze models now predict to drive content-aware cropping and retargeting. Seeing is an active, serial, attention-guided sampling process, not a parallel snapshot.

We feel we perceive a sharp, complete, stable visual world, but the signal reaching the brain is a ragged sliver: high-resolution color only at the fovea, a coarse statistical summary in the periphery, a blind spot, and a blanking on every saccade. The rich scene you seem to see is manufactured, filled in, and largely inferred — Hertzmann's illusion of awareness. Inattentional blindness (the invisible gorilla), change blindness (the door study), and crowding show how much we routinely miss. The lesson for imaging is double: seeing is not like taking a picture, so a photograph is not a transcript of experience; and the very inference and filling-in the brain performs is what makes perceptual shortcuts (compression, foveation, inpainting, saliency) work at all.

Photographers measure light in stops — a log₂ (doubling) scale. A stop is a log₂ unit: one stop is a factor of $2$ in light, and $\text{stops} = \log_2(\text{light ratio})$. Aperture (f-stops), shutter, ISO, and dynamic range are all counted in stops, so exposure is additive in log space — adding one stop here and subtracting one there keeps the total fixed, which is exactly what makes the three controls of the exposure triangle interchangeable. Watch the units, though: shutter and ISO step in factors of 2, but the f-number steps in factors of √2 — because the f-number scales the aperture diameter, while the light admitted scales with area $\propto$ diameter², so one stop of aperture runs f/2 → f/2.8 → f/4. It is the same "ratios matter, work in log" principle we meet again in gamma encoding and tone mapping.

When too few levels force you to round, rounding makes a correlated, structured error: a smooth gradient collapses into flat bands the eye locks onto as false contours. Dithering spends that same error as high-frequency (blue) noise instead, scattering it spatially (a randomized threshold, or error diffusion that pushes each pixel's residual onto its not-yet-visited neighbors) so the local average still tracks the true tone. It works because the visual system integrates noise but sees structure: it low-passes the grain back to the gradient while it would have flagged the bands. The same trade recurs wherever a few levels stand in for a continuum: halftoning in print, coded apertures / masks, computational illumination, blue-noise sampling in rendering, and stochastic rounding in low-precision arithmetic.

Walk an image the way it is laid out in memory. For row-major (HWC) storage that means the inner loop runs over $x$ (columns) and the outer over $y$ (rows) (for y { for x { … } }) so consecutive reads are contiguous, the cache line is used instead of thrashed, and the loop vectorizes. Get the nesting backwards and the identical arithmetic can run several times slower. The rule generalizes: make the innermost loop the fastest-varying index of your actual layout (planar adds channel outermost, for c { for y { for x } }; PyTorch's CHW differs again). It is the humble first instance of the book's performance theme (respect the memory hierarchy) and it returns in earnest in Performance engineering and Halide.

The numbers in an image array carry no meaning by themselves. The same triple (0.5, 0.2, 0.1) is a different color, and a different amount of light, depending on two things the array does not contain: - its color space — which primaries and white point those R, G, B refer to (sRGB? Adobe RGB? Display P3? a camera's native raw space?); and - its value encoding — how a stored number maps to light (linear, gamma, log, or a camera "look" curve). Strip those two away and you are left with a grid of anonymous numbers. Every operation that matters later (white balance, HDR merging, blur and deblur, any color conversion) silently assumes a particular answer to both questions, and gets the wrong result if the assumption is wrong. Hence the discipline of the whole book: never touch a pixel without knowing its color space and its value encoding. When two "identical" images don't match, this is almost always why.

Changing exposure is a multiplication of linearly-encoded values ($L_\text{out}=k\,L_\text{in}$, with $k=2^{s}$ for $s$ stops), equivalently an addition in log/stops ($\log L \mapsto \log L + \log k$). In gamma-encoded data it is neither, so the right move is to convert to linear (or log), apply, convert back. If you must work in place, the exact gamma-space map is still a multiply, but by $k^{1/\gamma}$ rather than $k$: $L_\text{out}=k^{1/\gamma}\,L_\text{in}$, the kind of off-by-a-gamma error that bites silently. This is the cleanest case of L23, "do the physics in linear light, apply gamma last."

A contrast change rotates values about a chosen pivot: $L_\text{out} = p + g\,(L_\text{in}-p)$. The pivot $p$ is exactly what is left unchanged, and it also decides whether the picture gets brighter or darker overall: a high pivot pulls most of the image down (darkening), a low pivot pushes it up (brightening). Unlike exposure, contrast is not radiometrically meaningful, it is a stylistic stretch, so the encoding matters less (no single space is "correct"); what matters is a sensible pivot, and the sensible pivot does depend on the space: roughly 18% mid-gray in linear light, the numeric midpoint in gamma or log. Pick the pivot deliberately: it is the whole character of the operation.

You do not need a clever spatial operator to fit a high-dynamic-range scene into a display: a smooth, monotone curve that compresses the highlights while preserving their local contrast (the Reinhard $L/(1+L)$ family, the photographer's S-curve, the filmic and ACES curves) goes a long way. It is the global tone-mapping workhorse: cheap, artifact-free, order-independent, and good enough that the local, edge-aware methods are the refinement, not the baseline. The recipe is just "be smooth, roll the highlights off gracefully, keep the shadows," and it manages most of the range most of the time.

A convolution replaces each pixel by a weighted combination of its neighbors, in practice usually a local one (a small window). Two things define it: the weights depend only on the relative position of each neighbor (not on where you are or what the values are), and they are the same everywhere in the image. That sameness is shift-invariance: slide the input and the output just slides to match. "Linear + shift-invariant + local" is the whole definition, and it is why a convolution is captured by one small kernel, why blur, gradients, and sharpening are all "one stencil applied everywhere," and why the next chapter's Fourier transform can diagonalize it.

Taking an image derivative is a convolution. With finite differences it is the obvious little stencil ([−1, 0, +1], or Sobel) and with the proper mathematical treatment a continuous derivative is convolution with the derivative of a smoothing kernel (the derivative-of-Gaussian). So gradients, edges, the Laplacian, and unsharp masking are all linear shift-invariant filters (L5.7), which is exactly why they have clean Fourier behavior, why edge detection is "just" filtering, and why combining a derivative with a blur (Sobel, derivative-of-Gaussian) is the natural noise-robust way to measure change.

Real optical blur is usually not one kernel convolved over the whole image: camera shake is an integral over homographies (its in-plane-rotation part reverses direction across the frame), defocus scales with 1/depth and breaks at occlusion boundaries, lens aberrations (coma, spherical, chromatic) worsen from center to edge, and motion blur follows each object's own motion; diffraction is the near-shift-invariant exception, spoiled only by aperture-clipping vignetting. In many cases the blur is locally close enough to a convolution that the LSI tools (Fourier, deconvolution) still apply patch by patch; in others you need spatially-varying kernels or fuller modeling (fake depth of field, camera-shake removal with in-plane rotations, inverting the optical PSF). The kernel may depend on 1/depth, local motion, and field position.

Express a linear operator in a basis where it becomes diagonal. A coupled, hard-to-invert operation in one representation can be a list of independent scalar multiplications in another, and scalars are trivial to understand, compose, and invert. Fourier diagonalizes convolution, turning a blur into per-frequency multiplication. This is not a one-off trick: it is the single most reusable move in applied linear algebra, and it returns for differential equations (the heat equation, below), for principal components, and for nearly every inverse problem in this book. The idea is already visible in the smallest possible case, a $2\times2$ matrix (a figure).

Because light is a wave, wave optics is naturally a story about sines and cosines, so Fourier analysis is usually the right language for it. The vivid everyday case: the star pattern thrown around a bright light when the lens is stopped down is, almost exactly, the Fourier transform of the aperture (its squared magnitude). A round opening gives the Airy disc and rings; a polygonal iris gives a sunstar whose every spike is the Fourier transform of one straight blade-edge, a perpendicular line of ripples, so the star reads out the shape of the diaphragm. The optics performs the transform for free, no FFT required; it is the same physics that makes a small aperture diffract (blur) more, and it returns as the lens's MTF (the transform of its PSF) in the OPTICS part.

Fourier diagonalizes convolution beautifully, but at two prices: it treats the signal as cyclic (wrapping at the edges, which real images do not), and its basis waves have infinite, global support (so an idealized frequency operation wants an infinitely wide spatial kernel). Real images are finite and demand compact-support operators. So the insights of Fourier (which frequencies are present, how invertible an operator is, where noise will blow up) are indispensable, but the implementation is almost always a compromise between the Fourier ideal and the reality of finite, non-cyclic images: windowing, padding, truncated kernels. Use Fourier to understand; use compact operators to compute.

In the Fourier basis a convolution is diagonal: it multiplies each frequency by one number (the kernel's response, the MTF), so undoing it is a per-frequency division. The whole stability question becomes local to each frequency: how near zero is that number? Blur is low-pass, so its response is tiny at exactly the high frequencies it destroyed; the lower a frequency's coefficient, the more its noise is amplified on inversion. That is why naive deblurring fails worst precisely where the blur was strongest, and why a Wiener / regularized filter stops dividing once the signal drops below the noise floor.

To represent a signal from its samples you must sample at more than twice its highest frequency (the Nyquist rate). Sample any slower and the frequencies above the Nyquist limit (half the sampling rate) cannot be stored: they fold down and alias onto lower frequencies, masquerading as structure that is not there (the moiré on a fine facade, the backward wagon-wheel). This is the spatial root lesson of all sampling, and it is not confined to space: the very same theorem governs the time axis, where under-sampling a fast motion gives temporal aliasing: the wagon-wheel and rotor-freeze effects of film and video. The only cure is to band-limit before you sample: low-pass (blur / area-average) away everything above Nyquist first, then sample (the anti-aliasing prefilter, L16). The frequency-domain account of why (sampling replicates the spectrum and the copies overlap) is L5.15; the ideal band-limiting filter, the unreachable sinc, is L5.16.

Sampling on a grid is multiplication by a comb (a train of deltas), and a comb's Fourier transform is another comb, so sampling replicates the signal's spectrum at every multiple of the sampling rate. If the spectrum is band-limited below Nyquist the copies sit side by side and the original is recoverable; if it is not, the copies overlap, and the folded-over tails land on top of the genuine low frequencies as aliasing: energy appearing at completely wrong frequencies (moiré, the backward wagon-wheel). The cure is to band-limit before sampling: low-pass away everything above Nyquist first, then sample.

Both jobs sampling demands, the anti-alias pre-filter (a perfect low-pass before sampling) and the reconstruction filter (a perfect low-pass to interpolate between samples), have the same mathematically ideal answer: the sinc, a perfect brick-wall box in frequency. But a vertical wall in frequency is an infinitely wide, forever-ringing kernel in space (the space–frequency tradeoff again), so the ideal is physically unreachable: you cannot store or convolve an infinite kernel, and truncating it makes it ring. Every real pre-filter and every real resampling kernel (box, bilinear, bicubic, Lanczos) is therefore a compromise away from the sinc, trading a little lost mid-frequency contrast against a little leaked aliasing. There is no free lunch and no perfect filter; choosing one is choosing which error to tolerate.

Fourier analysis says the perfect reconstruction and anti-aliasing filter is the sinc, a perfect box in frequency. But the sinc is infinitely wide and rings when truncated, so no real filter can be it. Every practical kernel (bilinear, bicubic, Lanczos) is a deliberate compromise between approximating that idealized sinc (what Fourier wants) and staying compactly supported (what computation, and ringing, demand): reach wider and you approach sinc; truncate harder and it gets cheaper and less ringy but further from ideal.

Read off the frequency response, the same compromise is a tug-of-war between two errors. Passband droop: a too-gentle filter attenuates frequencies below Nyquist that the grid could have represented, so the image goes soft. Stopband leak: a too-aggressive filter lets frequencies above Nyquist through, which alias into jaggies and moiré. A good kernel sits at the sweet spot (flat passband up to Nyquist, sharp rejection beyond) and box, tent, bicubic and Lanczos are just different points on this droop-versus-leak curve.

It is tempting (and a common mis-teaching) to picture "bicubic" as fitting a cubic polynomial through the nearby samples, a little least-squares regression per output pixel. It is nothing of the sort. Bicubic resampling is an ordinary convolution with a fixed, closed-form bicubic kernel $k_{BC}(x)$ (a member of the Mitchell–Netravali $(B,C)$ family), evaluated at the fractional offset each output sample lands on: no fitting, no solving, no per-pixel polynomial, just one continuous kernel dropped at a non-integer center and dotted against the neighbors. The same is true all the way down the menu: bilinear is convolution with a tent, Lanczos with a windowed sinc, nearest-neighbor with a box. Every "interpolation" here is a convolution; only the kernel changes.

The Fourier chapter's lesson was diagonalize when you can: work in the basis where your operator is simple. This chapter adds the practical sequel: the perfect basis is rarely the useful one. Fourier is perfectly localized in frequency and not at all in space; the primal grid is the reverse. The uncertainty principle (compact in space ⇔ spread in frequency, from the Fourier chapter) says you cannot have both perfectly. A pyramid surrenders a little of each (it chops frequency into a few coarse bands and keeps each band laid out spatially), and that modest sacrifice is exactly what makes per-scale, per-region editing possible. Reach for a representation that is local in both, not for the one that is perfect in one.

The squared-error (L2) loss dominates engineering and machine learning not because it tracks how images look (it does not) but for two unglamorous reasons. It penalizes big errors disproportionately: squaring an error of 2 counts four times as much as an error of 1, so the optimizer spends its effort flattening the worst mistakes (a feature when outliers are the enemy, a bug when a few spiky pixels hijack the fit, which is when L2.6 / mean-absolute error is more robust). And it is trivial to optimize: the derivative of a quadratic is linear, so its gradient is just the residual and the minimum is a linear system (the normal equations) solvable in closed form, the engine of least squares and the cheap, smooth backprop target of choice. Reach for L3.8 with eyes open: you are buying differentiability and outlier-sensitivity, not perceptual fidelity. Its perceptual cost has its own lesson: the L2-optimal answer is the dull average (Big lesson L5.24, in Denoising basics).

Every quality metric here is fair game as a training loss, until you actually train against it, at which point the network learns to exploit its blind spots rather than improve the underlying quality, and the metric quietly stops measuring what you wanted. This is Goodhart's law (Charles Goodhart, 1975, on monetary targets; sharpened by Marilyn Strathern, 1997, to "when a measure becomes a target, it ceases to be a good measure"), with Campbell's law its social-science cousin. In imaging it bites constantly: optimize for PSNR and you get oversmoothed, MSE-hedged mush; optimize for SSIM and textures get gamed; optimize against a fixed LPIPS network and you find its adversarial soft spots. The discipline: keep your evaluation metric separate from your training loss, hold out a metric (or, better, a human study) you never optimized against, because any number you optimize hard enough ceases to certify the thing it once measured.

The move that makes the bilateral filter work recurs throughout the book, so name it. Use the color / intensity difference between two pixels as a measure of how much they "belong together," their affinity. Pixels with high affinity are treated as measurements of the same underlying thing and get averaged; pixels with low affinity (across an edge) are kept apart. The bilateral's range weight $g$ is the first instance of an affinity: a similarity computed from a value difference. Once you see denoising this way, the question "which neighbors count as the same value?" has a clean answer (the ones with high affinity) and the same affinity idea will go on to drive edge-aware tone mapping (the halo fix), edge-aware selections, joint / cross filtering, the bilateral grid, the guided filter, non-local means, colorization, matting and segmentation. We register the lesson here, in denoising, where it first earns its keep; the full edge-preserving treatment (the family of methods and the optimization form) is the subject of the EDGES MATTER part. Edge-preserving is affinity.

Minimizing expected squared error has an exact and revealing answer: the mean of all the outcomes consistent with the data. When the answer is genuinely uncertain (what the denoised patch really looked like, which color was under that gray pixel, what texture filled that hole) the $L_2$-optimal estimate hedges by averaging over all the possibilities, and the average of many sharp, plausible answers is one blurry, desaturated, washed-out answer. This is why $L_2$ denoisers oversmooth, why $L_2$ colourization comes out muddy sepia, why $L_2$ super-resolution is soft: the loss is doing exactly its job, and its job is to play it safe. It is the perceptual flip side of why $L_2$ is so convenient in the first place (Big lesson L5.21, in Image metrics). Escaping the dullness means changing the objective: an adversarial or perceptual loss that rewards a plausible answer over the average one, or a generative model that samples a single sharp outcome instead of averaging them all (the diffusion connection in the sidebar above). And do not then turn around and score the sharper result with the very $L_2$ you abandoned (Big lesson L5.22).

Edge-directed demosaicking is a first, very concrete instance of a theme that returns throughout the book: adapt the operation to the local structure of the image rather than applying the same fixed filter everywhere. Here the structure is an edge and the adaptation is choosing the interpolation direction; later, in edge-preserving filtering and denoising, the same instinct (use the similarity between pixels to decide how much they should influence each other) becomes the bilateral filter and the idea of affinity. Whenever a fixed linear filter blurs across something it should have respected, the cure is to look at the data first.

This is the same decomposition that runs through the whole book: split an image into a detail-rich part and a slowly-varying part, and treat each appropriately. Here detail lives in green (a stand-in for luminance) and is interpolated carefully; color lives in the differences $R - G$, $B - G$ (a stand-in for chrominance) and is so smooth it can be interpolated crudely. It is the same perceptual fact that lets JPEG subsample chroma and lets us denoise color more aggressively than brightness: luminance carries the acuity, chrominance can be coarse.

The inverse problems of this part produce a system $A^\top A\,x = A^\top y$ that is far too large to form or invert directly, but is sparse and structured — $A$ is a convolution. So never build the matrix: solve iteratively, touching $A$ only through apply-operator calls (a forward convolution and its transpose). Plain gradient descent zig-zags on the elongated, ill-conditioned bowl; conjugate gradient picks non-interfering directions and converges in tens of steps, not thousands; a preconditioner $M\approx(A^\top A)^{-1}$ re-rounds the bowl and slashes the count further. This same matrix-free spine — set up $y=Ax$, minimize $\lVert Ax-y\rVert^2$, solve with CG / multigrid / FFT — drives Poisson image editing, colorization, matting, and gradient-domain HDR.

A classical recovery method minimizes data-fit + a hand-tuned prior — smoothness, sparsity, a dark-channel heuristic, whatever a person decided made a good image. A learned method keeps the same skeleton but replaces that prior, or the whole operator, with a function $f_\theta$ fit to a dataset. The inverse-problem template does not change; the prior simply becomes data-driven. This is a throughline for the rest of the book: the deep-network realizations follow in Deep learning, a learned denoiser turns out to be a reusable prior you can plug into any solver (Denoising as a universal prior), and a diffusion model is the same idea taken to its generative limit (Generative AI and diffusion). The cost is real and worth naming up front — you now need data and compute, and the prior can hallucinate, inventing plausible detail that was never measured. (First appearance; the refresher carries a one-line callback. Big Lessons)

Every prior in the book so far was one you could only evaluate: a penalty $\Phi(x)$ you add to a data term, or a denoiser you plug into a solver. A generative model is a prior you can draw fresh images from, $x \sim p(x)$. That leap — from scoring an image to sampling one — is the whole generative idea, and it makes three things possible at once: unconditional generation (an image from nothing), conditioning (text→image, image→image — sampling from $p(x \mid c)$), and posterior sampling for inverse problems (condition the prior on a measurement $y$ → super-resolution, deblurring, inpainting). The data-fit-plus-prior skeleton is unchanged; the prior merely learned to generate. This is L6.3's first appearance; registered as L6.3.

When the measurement genuinely destroys information — super-resolution past the sensor's sampling, deblurring frequencies the blur erased, inpainting a hole — only a prior can select an answer L6.4. A generative model is that prior at full strength, and conditioning it on the measurement is posterior sampling. (First appears in Super-resolution; see L6.4.)

The bilateral is awkward because it is non-linear: its weights depend on the pixel values, so they differ at every output pixel and the filter is not a convolution. The bilateral grid's deeper trick, beyond "add a dimension", is to discretize the intensity (range) axis into a small set of levels, and then do cheap, ordinary linear work at each level. Quantize $I$ into grid layers, run a plain blur within and across layers (a value never mixes with one in a distant layer, so edges are preserved for free), and interpolate back to each pixel's exact value. The per-pixel nonlinearity becomes "a few linear passes plus a lookup." This is a general recipe, not a bilateral-only hack: the fast local Laplacian filter (Local Laplacian filters, Aubry et al. 2014) accelerates its per-pixel nonlinearity the same way: evaluate a handful of discrete intensity levels of a slow operator and interpolate between them. Whenever a filter's cost comes from a continuous, value-dependent response, discretizing that response axis turns it into reusable, parallel, convolution-shaped work. (Registered in Big Lessons as L7.1; recurs in Local Laplacian filters and, as an algorithm-level speedup, in Algorithmic speedups.)

Lifting the image into the higher-dimensional grid $(x, y, I)$ is what makes the coarse grid safe. Because pixels on opposite sides of an edge sit at different intensity heights, they are already separated along the range axis, so you can downsample the spatial axes aggressively without ever blurring across an edge, which the spatial downsampling never touches. Two payoffs follow: high-resolution edges survive despite a coarse grid; and a large spatial kernel becomes a small blur on the coarse grid, so the bilateral filter grows cheaper as its spatial extent grows, the opposite of an ordinary convolution. Paying one extra dimension for cheap, edge-aware, large-scale smoothing is a trade that recurs: NL-means' patch space, the permutohedral lattice, and the learned bilateral grid (HDRnet) below all live on it.

The eye cares about local contrast, not absolute values, so edit the contrast directly, in the multiscale decomposition. Where gradient-domain methods (Poisson image editing) manipulate the gradient field and re-integrate, local Laplacian filtering manipulates the Laplacian-pyramid coefficients (the local contrast at each scale) with a per-pixel point curve, then collapses the pyramid. Both discard or reshape contrast while letting the absolute level fall out, and both are kin to the bilateral base/detail split; the difference is that working band by band with an identity tail for large contrasts makes the result halo-free by construction, abolishing rather than merely reducing the artifact that motivated the bilateral. (L7.3, registered in Poisson image editing; this is its multiscale-decomposition appearance.)

Many image edits are discrete optimization on a graph or grid, whether dynamic programming (a least-cost path), min-cut / max-flow (a globally optimal boundary), or a spectral relaxation (normalized cuts), and the energy (affinity) you choose is the whole game. The optimizer is generic and off the shelf; what you penalize (gradient magnitude along a seam, label disagreement across an edge, the cost of severing two regions) is the entire design. Read this as the L5.23 affinity principle from the bilateral chapter turned inside out. There, the affinity between two pixels said "average these together." Here, the very same quantity, sitting on a graph edge, says "do not cut here." High affinity (similar neighbors) marks an expensive place to cut; low affinity (a genuine edge) marks a cheap one. Design that cost well and a generic min-cut returns the boundary a person would have drawn by hand. (Registered as new lesson L7.4 · image edits as discrete optimization on a graph (the energy is the design), first appearance here. Extends L5.23. Also relevant downstream: graph-cut photomontage composed with a Poisson blend; compositing / segmentation / matting; texture synthesis; stereo and Markov-random-field labeling.)

A free-form warp that only interpolates handle positions will happily shear, stretch, and collapse a shape, because position-interpolation has no opinion about how a region deforms, only about where its anchors land. To make a deformation look physical you add a regularizer that measures, at every point, how far the local transformation is from a rigid motion (a rotation + translation) and minimizes it. Rotation is left free (a limb may swing any amount); only the stretch/shear part is charged. The recurring move (fit the constraints, but regularize toward a prior about the world) is the same one behind Poisson editing (regularize toward a gradient field), content-aware retargeting (regularize toward leaving salient regions rigid), and morphable models (regularize toward a learned shape basis). Here the prior is the most generic physical one there is: things rotate freely but resist stretching.

The same $2\times2$ matrix $M=\sum_W \nabla I\,\nabla I^{\top}$ answers three questions across this part, and the unity is the point. Is this a good point to detect? Yes, where $M$ has two large eigenvalues, a corner: that is Harris / Shi–Tomasi, here. Is this patch's motion solve well-conditioned? The very same matrix, now required invertible: that is KLT (Feature tracking). What is the local flow constraint? — $M$ is the normal-equations matrix $A^\top A$: that is Lucas–Kanade (Optical flow). Detect with it, track with it, solve flow with it — one matrix, read three ways. Corners are special for a deep reason: the same both-directions-vary condition that localises a point is also what makes its motion recoverable. So "good to detect" and "good to track" are not two coincidentally-aligned criteria; they are one statement about $M$.

Whichever operator you pick, do the range compression in log luminance, not linear. Dynamic range is multiplicative — light spans ratios (stops, L4.1), and the eye reads ratios — so in log a fixed ratio of contrast becomes a fixed additive step (equal perceptual contrast gets equal treatment), an exposure change becomes a constant shift (L5.4), and every method behaves: the global curve rolls off evenly across stops, the gradient attenuation shrinks multiplicative edges uniformly (L7.3), the base/detail split adds a clean log-detail layer back. Compress in linear light instead and the same curve crushes the shadows and warps local contrast, because equal linear steps are wildly unequal ratios. Convert back to display-linear only at the very end. (L10.1, the tone-mapping face of the encoding lesson L3.8; the H&D curve below is plotted against log exposure for exactly this reason.)

Whenever one photograph is bounded along some axis — too little dynamic range, too narrow a field of view, too shallow a depth of field, too much noise, too few color bands — the remedy has a single shape: don't fight the limit inside one frame, capture several frames that each commit differently along that axis, then combine them into one image that exceeds what any single exposure could hold. HDR beats the sensor's dynamic range; panoramas beat the lens's field of view; focal stacks beat depth of field; burst averaging beats noise; hyperspectral beats three color channels. It is the close cousin of L12.2 above (capture the full set, decide later): L12.2 is about deferring the decision, L12.1 about exceeding the medium's ceiling.

The usually-forgotten thesis of Adelson & Bergen's plenoptic paper (→ Image measurements as integrals) is that low-level imaging does one kind of thing, repeated across every dimension — space, time, aperture, wavelength. This chapter is that thesis turned into engineering. Coding the exposure in time (the flutter shutter, the parabolic sweep) is the same move as coding the aperture in space (Coded aperture), as the phase plate of Wavefront coding, as focal-sweep extended depth of field, as light-field refocusing across the aperture, as chroma tricks across wavelength. Sample-or-integrate, code-or-leave-clear, make-the-blur-invertible-or-invariant: pick a slice of the plenoptic function and the toolkit is the same. So when a new imaging problem arrives, ask which axis it lives on and which of these moves applies — the answer is usually already in the book, sitting on a different axis. (Light field across the aperture is what high-speed video is across time; a coded aperture is what a flutter shutter is in time.)

L2.43 set the bottom of a single exposure at the read-noise floor — the dimmest signal that survives the amplifier and ADC. But that floor is an engineering limit, not a physical one. Drive a photodiode past breakdown (a Geiger-mode avalanche) and a single photo-electron makes a macroscopic, digitally-countable click: the floor drops to one photon, read noise effectively zero, and the only noise left is the shot noise of the light itself (L2.35). So dynamic range is bounded at the top by full-well capacity and at the bottom, ultimately, by the discreteness of light, not by the chip. Single-photon sensors live at that opposite extreme: where an ordinary pixel sees only its own read noise, they count individual arrivals.

Velocity in a video is almost always inferred: track a feature or solve optical flow across two frames and divide displacement by time, producing an estimate that needs texture, good light, and an aperture-problem-free edge, and that arrives a frame late. A Doppler sensor skips the inference and measures radial velocity directly, in one instant, from the frequency shift the moving surface imprints on a coherent (laser/radar) return. The payoff is the usual one for a direct measurement over an inferred one: it is fast (no two-frame baseline), works in low light and on textureless surfaces (it rides the active illumination, not scene contrast), and it reads the component optical flow cannot, motion along the line of sight. The blind spot is the mirror image: it is deaf to motion purely across the frame, which is exactly optical flow's strength.

A conventional sensor is synchronous and absolute: every pixel reports its brightness, on a shared clock, every frame, including most pixels that did not change, paying full data, power, and latency to re-send a static world, and smearing whatever moved into motion blur. An event sensor (and, in its own way, the single-photon sensor) inverts this: each pixel is asynchronous and differential, staying silent until something happens and then firing a timestamped message the instant it does. The result is structural: microsecond latency, no motion blur, huge dynamic range, and data/power that scale with scene activity, not pixel count. The cost is equally structural: there is no frame and no absolute intensity, so the entire downstream toolkit, built on dense, synchronous arrays, must be rethought for a sparse, asynchronous stream.

The recurring move of making a good picture is separation: lift the subject out of its background so the eye knows, instantly and without effort, what to look at. You have four levers to do it — depth of field (throw the background out of focus), framing and viewpoint (find a clean background, build layers front to back), light (a rim light, a pool of brightness, contrast against shadow), and color (a warm subject on a cool ground, or a saturated subject on a desaturated one). Pick a lever — and then commit. Whatever you do — off-center the subject, blur the background, tilt the horizon, push the saturation — do it boldly or not at all. The timid half-measure, the subject that is almost off-center, the background that is sort of blurred, is the amateur's tell. Most of a photograph's craft is deciding what to leave out, and then leaving it out without flinching. Every rule in this chapter is one tactic in service of this single idea.

On a modern machine a multiply-add costs ~1 cycle while a main-memory access costs ~200, and moving a word across the chip costs orders of magnitude more energy than the arithmetic done on it. So "make it faster" almost never means "do less math" — it means move fewer bytes: improve locality so data is reused while it is close, and fuse operations so intermediates never round-trip to memory. This is the memory wall, and it is the organizing fact of the whole part. The corollary you can act on today: when you profile slow image code, suspect the data movement first and the arithmetic second.

The highest-leverage transform in image code is tiling (blocking): restructure the loops so the data in flight at once — the working set — fits in a fast cache, and so every byte you load is reused many times before it is evicted, instead of round-tripping to DRAM between stages. A separable blur, a pyramid level, a stencil, a two-pass filter — all run dramatically faster tiled, with identical arithmetic, purely by improving locality. Tiling is to the memory hierarchy what a good night's packing is to a small suitcase: the contents are the same; only the data movement changes.

Two laws frame what parallelism can and cannot buy. Amdahl's law is the pessimist: if a fraction $s$ of your program is irreducibly serial, the speedup from $N$ parallel workers is $1/\!\left(s + (1-s)/N\right)$, which is capped at $1/s$ no matter how large $N$ grows — a program that is 5% serial can never run more than 20× faster, even on infinite cores. Gustafson's law is the optimist's correction: in practice we do not hold the problem fixed and add cores — we grow the problem (bigger images, more frames, higher resolution) so that the parallel part dominates and the serial overhead becomes negligible. Both are true. Amdahl warns you to hunt down serial bottlenecks; Gustafson reminds you that the workloads we actually care about scale.

Two separable decisions set the speed of an imaging kernel. The algorithm is what you compute — the mathematical operations, the FLOP count. The schedule is how you compute it — the loop order, the tiling, the vectorization, the fusion, the parallelization — and it is the schedule, not the algorithm, that determines how much data moves and therefore, on a bandwidth-bound machine, how fast the code runs. The same algorithm can run an order of magnitude faster or slower depending only on its schedule. Recognizing that these are independent choices is the conceptual key to the whole part: it is exactly what Halide: Decoupling Algorithms from Schedules makes explicit and retargetable, letting one algorithm be re-scheduled for a CPU, a GPU, or a phone without rewriting the math.

Imaging hardware lies on a single spectrum that trades flexibility for efficiency. The CPU runs any code at the lowest efficiency; the GPU gives up generality for data-parallel throughput; the NPU/DSP specialize to tensor math or real-time signal loops for more efficiency still; the ISP/ASIC give up programmability altogether for the highest performance-per-watt. No single unit wins everywhere, so a real camera is a heterogeneous SoC that places each stage on the unit that fits it, and the engineering is orchestrating the whole machine and minimizing the data moved between its parts — not optimizing one core. Where you put a computation on this line is a design decision as consequential as the algorithm itself.