• three models: **ray** (geometry), **wave** (color, diffraction), **particle/photon** (counting → noise) — use whichever fits the question [wave/ray/particle figure]

• side bar — **wave–particle duality, made visible with a liquid analog**: light is *both* wave and particle, which is famously hard to picture. A startling **hydrodynamic analog** makes the marriage tangible: a tiny oil droplet **bouncing** on a vibrating fluid bath becomes a **"walker,"** carried across the surface by its own **pilot wave**, and these walkers reproduce quantum-like behaviour (single- and double-slit statistics, quantized orbits, tunnelling). They are *not* a theory of quantum mechanics, but they are a gorgeous classical picture of a particle that is inseparable from its wave. [embed video — Couder & Fort and **John Bush's** (MIT) walking-droplet experiments → [[Video Resources]]] (revisited at diffraction, below)

• wavelength, EM map, spectra

• **Newton's prism experiment** (the historical hook): a prism splits white light into the **spectrum** — white light is a *mixture* of wavelengths, and the *experimentum crucis* (a second prism) shows those pure colors don't split further. [classic diagram]

• a spectrum = power vs wavelength; like an **infinity of channels**, of which a camera (and the eye) keep only three → **metamerism** (Color technology)

• a camera sees nothing that hasn't bounced off something

• **reflection** (angle out = angle in); **refraction** (light slows in glass/water → *refractive index*; **Snell's law** n₁ sinθ₁ = n₂ sinθ₂); **diffraction** (→ Wave effects section below)

• marching-band (wavefront) intuition; fuller lens geometry deferred to Single lenses

• **interactive — a wave simulation of refraction** (`fig-refraction-wave-sim`): the same 2-D FDTD wave solver as the diffraction figure, but with a slower medium (higher index) filling the lower half of the box. An angled plane wave crosses the interface and its wavefronts **bend toward the normal** and **crowd together** (λ→λ/n) — the wave speed drops, so Snell's law becomes a *consequence* of the wave picture (the wave-level version of the marching-band intuition and Fermat's equal-path view in [[#Single lenses: refraction and Snell's law]]). Sliders for the index n₂ and the incidence angle, with an on-figure Snell check. Companion to `fig-diffraction-wave-sim`.

• **light–matter interaction is largely *resonance***: much of what light does to matter — the **index of refraction** itself, and **absorption** (hence material color) — comes from the **resonances** of atoms, molecules, and bound electrons. Light's oscillating field drives those bound charges, which resonate at characteristic frequencies (the **Lorentz-oscillator** picture); **near a resonance** the material **absorbs** strongly, and the **index of refraction varies with wavelength** (**dispersion**) according to how far each wavelength sits from the nearest resonance. This is why different materials transmit/absorb different colors, and why $n(\lambda)$ gives **prisms, rainbows, and chromatic aberration**. Flag forward: dispersion **shapes much of lens design** — achromatic/apochromatic doublets (crown+flint), the fight against chromatic aberration in fast lenses → [[Optics, lenses, and aberration correction]]. **Interactive** `fig-lorentz-resonance`: a driven bound electron (mass on a spring) with the absorption peak at ω₀ and the n(ω) normal→anomalous dispersion curve; sliders for driving frequency and damping.

• side bar: **polarization** — light is a *transverse* wave (field ⊥ propagation); **polarization** is the **orientation of that oscillation**. Natural light is **unpolarized** (orientation a rapidly varying mix, *no single direction*); reflection partly polarizes it → a polarizing filter cuts reflections / deepens the sky. Two everyday encounters: (a) **good sunglasses are polarizers** that selectively kill glare — *why* lives in the reflection/BRDF material (specular reflection off dielectrics is partially polarized, maximally at **Brewster's angle**) → [[#The BRDF and the look of materials]], the polarizer filter in [[#Lens filters: polarizers, ND, and graduated ND]], and reflection / specular removal in [[Single-image computational photography]]; (b) **LCD displays emit polarized light** — the liquid-crystal shutter forms an image by **rotating polarization between two crossed polarizers**, which is why a polarizer (or polarized sunglasses) makes some screens go dark at certain angles.

• **why some screens go dark through sunglasses and some don't.** Polarized sunglasses pass only **vertically** polarized light (so they can block the horizontally-polarized glare off roads and water). An LCD emits **linearly polarized** light fixed by its **exit polarizer**; whether the screen survives your sunglasses depends on *that polarizer's orientation*: if it emits **vertical** light it passes straight through, if **horizontal** the glasses **black it out**, and at **45°** it dims. Manufacturers choose the axis — laptops and monitors are usually safe in landscape, but many phones (and especially **car dashboards / GPS**, ATMs, and aircraft seat-back screens) are built with a diagonal or horizontal axis, so they wash out or go black when you tilt your head or rotate the phone 90°. The honest fix is to put a **quarter-wave retarder** *outside* the exit polarizer so the emitted light is **circularly** (not linearly) polarized — then it looks the same through any linear polarizer at any rotation; phones that stay bright through sunglasses generally do this.

• side bar: **why the sky is blue and the sunset orange** — **Rayleigh scattering**: air molecules scatter short (blue) wavelengths far more than long (red) ones (∝ 1/λ⁴). At midday the sunlight takes a *short* path through the atmosphere and the blue it sheds scatters into the sky in every direction → the sky reads **blue**; at sunset the light skims a *long* path, its blue is scattered away before it reaches us, and mostly **orange–red** survives → the sun reddens. (Larger droplets scatter all wavelengths about equally — **Mie** scattering → white haze and clouds.) [atmospheric scattering figure] [Minnaert]

• side bar: **rainbows** (spoiler: it's a caustic) — and a desktop demo. A **single spherical raindrop** bends sunlight three times: **refract** in at the front, **reflect** off the back, **refract** out again; because $n$ depends on wavelength (**dispersion**, above) each color exits at a slightly different angle. Across a whole sky of drops those exit rays **pile up** into a bright **caustic** at the **rainbow angle** — about **42° (red)** down to **~40° (violet)** measured from the **antisolar point** (the shadow of your head) — and *that* accumulation is the rainbow. A **glass sphere** reproduces it on a tabletop: it **stands in for a giant water droplet**, and a bright lamp throws its dispersed caustic onto a card (`fig-rainbow-glass-sphere`).

• **the color order, and the second bow**: the **primary** bow has **one** internal reflection — **red on the outside, violet inside**. A fainter **secondary** bow (~51°) comes from **two** internal reflections, with the colors **reversed**; between them sits **Alexander's dark band** (named for Alexander of Aphrodisias, ~200 AD), darker because one-reflection light emerges only at **≤42°** and two-reflection light only at **≥51°**, so the 42–51° gap gets almost no rainbow light (and the sky *inside* the primary is correspondingly brighter). Same geometry, one extra bounce. Real example: a **double bow over Loch Ness** (`fig-rainbow-loch-ness`, photo) showing both bows and the dark band.

• **dispersion alone is NOT a rainbow**: a prism only *fans* colours; the rainbow's brightness and fixed angle are a **caustic**. Vary the impact parameter and the deviation $D(b)$ of the **one-internal-reflection** ray bottoms out at a **minimum** (~138° → bow at 42°) where rays pile up. Crucially, a ray with **only two refractions and no internal reflection makes no bow** — its $D(b)$ is **monotonic**, no minimum, no pile-up. Dispersion only splits the already-bunched light into the band. Interactive Descartes diagram `fig-rainbow-droplet` (water drop; drag the impact parameter, choose the number of internal reflections, and tick **"zoom near the caustic"** to re-range both the slider and the plot tightly around the bow; the deviation-vs-$b$ plot marks the minimum, and a side panel plots the **ray density** — an integral over impact parameter — whose **spike** at the stationary deviation *is* the bow. Dispersion is always on, so three colour bands separate near the caustic).

• **it's an angle, not an object**: the bow is centered on the antisolar point, so it can't be reached and *everyone sees their own* — it is a **caustic of accumulated rays at a particular angle**, not a thing sitting in space (the same "it's a direction" lesson as vanishing points). [Minnaert] [`fig-rainbow-glass-sphere`, real demo photo; `fig-rainbow-droplet`, interactive]

• **scattering & absorption** in media (fog, haze, underwater): single vs. multiple scattering → de-weathering / dehazing; the same physics as the blue sky, at close range

• a **red apple is not red in the dark**: color = **illumination × reflectance**

• reflectance ρ(λ) is the surface's intrinsic, fixed signature; illumination S(λ) is whatever light is present; their per-wavelength product C(λ) reaches the eye

• this multiplicative entanglement is exactly what **white balance** must later undo

• scope caveat: the clean product assumes direction-independent reflectance (matte case); the general directional case is the **BRDF**, next

• side bar: **albedo can appear > 100% (fluorescence cheats)** — real (non-fluorescent) reflectance is **≤ 1 per wavelength** (a surface can't return more light than it receives at that wavelength). But a **fluorescent** material **absorbs** short-wavelength (often **ultraviolet**) light and **re-emits** it at **longer** wavelengths, so in the re-emitted band the measured "reflectance" **exceeds 1** — apparent albedo tops 100%. Examples: **optical brighteners** in laundry detergent and paper ("whiter than white"), **highlighter** ink, **fluorescent safety vests**. The trick is **moving energy between wavelengths**, not creating it.

• the **BRDF** (bidirectional reflectance distribution function): a surface re-emits incoming light into a *distribution* of outgoing directions — radiance toward q = f_r(p→q; n, λ) · incoming; *bidirectional* = depends on both the incoming and outgoing directions. [BRDF diagram]

• **diffuse (Lambertian) vs specular (shiny)**: a matte surface scatters into a broad lobe and looks equally bright from any view (the light it *receives* falls off as cos θ); a shiny one concentrates near the mirror direction, giving a view-dependent highlight; most materials are a blend. [Lambertian sphere vs shiny sphere]

• **side bar — where a material's colour actually comes from.** "Colour" is not one mechanism; the same red can be made several physically different ways, and they behave differently (some shift with viewing angle, some don't), which is why this belongs with the BRDF:

• **absorption (pigments & dyes)** — the everyday case: molecules/electrons absorb certain wavelengths (the **resonance** picture from *Light and physics*) and what's left is the colour. Paint, ink, skin, leaves (chlorophyll), most natural surfaces. **Angle-independent** and the basis of the *illumination × reflectance* model above.

• **structural colour (interference & diffraction)** — colour from **micro-geometry**, not pigment: **thin-film interference** (soap bubbles, oil slicks, anodized metal, the blue of a Morpho butterfly, peacock feathers), **diffraction gratings** (CD/DVD, holographic foil, some beetles), and **photonic crystals** (opal). The hallmark is **iridescence** — the hue **shifts with viewing/lighting angle** — because it comes from path-length differences, so it is strongly **directional** and a genuinely *spectral* BRDF effect, not a single albedo.

• **scattering** — wavelength-dependent **Rayleigh/Mie/Tyndall** scattering makes colour without any pigment: the **blue sky** and blue eyes, the **Tyndall blue** of some bird feathers and milk, the red of a sunset (cross-ref the Rayleigh sidebar in *Reflection, refraction*).

• **metallic reflection** — in metals the **free electrons** set the reflectance; most metals are grey/white (aluminium, silver), but **gold** and **copper** absorb in the blue and so reflect warm — a coloured *specular* with a tinted highlight (unlike dielectrics, whose highlight stays the light's colour). Matters for shading models (metal vs dielectric is the key axis in PBR).

• **emission & fluorescence** — the surface **makes** light: incandescence/LEDs (emit), and **fluorescence/phosphorescence** which absorb short λ and re-emit longer (the albedo->100% and "fluorescence breaks the BRDF" sidebars).

• takeaway: pigment colour is a per-wavelength **albedo** (the simple model), but interference, scattering, and metallic colour are **angle- and spectrum-dependent**, which is precisely why a faithful material model is a *spectral, directional* BRDF rather than a single RGB reflectance. [PBR: Pharr, Jakob & Humphreys]

• **light–matter interaction also polarizes the light**: reflection (especially the **specular** component) partially **polarizes** what bounces off a surface — maximally near **Brewster's angle** — so a full account of the BRDF is *polarization-dependent* (a **polarization BRDF / pBRDF**). This is exactly what a **polarizing filter** exploits to cut glare and reflections (→ Photography 101 — Lens filters) and what **shape-from-polarization** and reflection-removal methods use (→ Advanced). [cross-ref the polarization sidebar in *Reflection*]

• **side bar — why good sunglasses are polarizers.** Glare off roads, water, snow, and car hoods is light that reflected off a roughly **horizontal** surface, so (near Brewster's angle) it is strongly **horizontally polarized**. Quality sunglasses are **linear polarizers oriented to pass vertical and block horizontal** light, so they kill that glare specifically — not just dim everything like a plain neutral-density tint. That's the difference between "good" (polarized) and cheap (just dark) sunglasses: you see *into* the water and the wet road instead of at the sheen. The same physics is the camera's circular/linear polarizer, and the same selective extinction is shape-from-polarization's signal. (One catch: polarized glasses can make some **LCD screens / phone displays** go dark at certain angles, because those screens emit polarized light.)

• **try it yourself (how to test your sunglasses).** Hold the glasses up to a glary reflection — a car hood, a puddle, a window, a lake — and **rotate them ~90°**. If the glare **darkens in one orientation and comes back in the other**, they are genuine **polarizers**; if nothing changes, they are just tinted. (That is exactly what `fig-polarizer-sunglasses` shows. A second quick test: look at an LCD screen and tilt your head — polarized lenses go dark at some angles.)

• hint of **global illumination** (light bounces many times between surfaces) — and it is not just a rendering nicety, it is something the photographer actively **exploits** and occasionally **fights**:

• **handy — reflectors and bounce flash**: because light keeps bouncing, you can shape it by adding bounce surfaces. A **reflector** (a white/silver/gold panel) catches the key light and **bounces fill** into the shadows; a **bounce flash** fires at a ceiling or wall so the *whole room* becomes a big, soft secondary source instead of a hard on-axis point. Both are global illumination put to work — softening contrast and wrapping light around the subject (→ Photography 101 — fill & bounce flash; [[#Human factors and the art of photography]]; computational bounce flash in Computational illumination). [photo: someone holding a reflector to bounce fill onto a subject — `fig-reflector-fill`, **shoot our own** unless a CC image turns up]

• **a challenge — unwanted bounced color**: the same bounced light can **hurt** you, because it carries the *color* of whatever it bounced off (color bleeding — the very thing the `fig-global-illumination` figure shows). The classic case is shooting at a **pool**: the strong **blue** light bouncing up off the water casts a cold tint on faces that makes decent **skin tones** genuinely hard to hold (a green lawn or a red wall does the same in its own hue). Knowing the bounce is colored is half the fix — block it, add neutral fill, or correct it later.

• **side bar — fluorescence breaks the BRDF.** A BRDF is **per-wavelength**: same wavelength in, same wavelength out, just redistributed in direction. **Fluorescence** violates this — it **absorbs short-λ (often UV) light and re-emits at *longer* wavelengths** (the **Stokes shift**), moving energy *between* wavelengths. So a fluorescent material is not described by a scalar BRDF per λ but by a **reradiation (bispectral) matrix** — input wavelength → output wavelength. (This is also the cheat behind apparent albedo > 100% — cross-ref, don't re-derive, the albedo sidebar in [[#The color of objects: illumination times reflectance]].) Everywhere once you look: **optical brighteners**, **highlighter** ink, **minerals under UV**, **GFP** in bio-imaging, **banknote security inks** — and it is the working signal in fluorescence **microscopy** and **forensics**.

• **side bar — subsurface scattering (why skin isn't plastic).** A BRDF assumes light leaves at the *same point* it entered. In **translucent** materials light instead **enters, scatters around beneath the surface, and exits somewhere else**, so a point-wise BRDF is **insufficient** — you need the **BSSRDF** (bidirectional *surface-scattering* RDF), which couples an entry point to a separate exit point. Examples: **skin**, **marble**, **milk**, **wax**, **leaves**. This is what gives skin its soft, "lit-from-within" glow and rounded shadow terminator; render skin with a pure BRDF and it looks **plastic**. (Cross-ref translucency / global illumination in rendering.)

• 'white' light is not one thing — the same scene at noon vs sunset shifts color

• **blackbody** & **color temperature** (low K warm/orange, high K cool/blue); **D65**, **tungsten** (warm), **fluorescent** (spiky → poor color rendition)

• insist again: the stimulus is the **spectral product** (illuminant × reflectance) → motivates white balance

• **radiance** L: power per unit area per unit solid angle carried *along a ray* — ultimately what a pixel measures; **conserved along a ray** in vacuum (a wall doesn't get dimmer as you step back, only smaller)

• **irradiance** E: power per unit area *arriving on a surface* (the sensor) — the integral of incoming radiance over the collecting solid angle, which grows with **aperture area**

• the **cos θ** in E = ∫ L cos θ dω is **projected area**: a beam (cylinder of light) of width w striking a surface at angle θ spreads over a larger footprint w/cos θ, so the same power is thinned → E ∝ cos θ (Lambert's cosine law). [cosine / projected-area figure]

• **sidebar — seasons**: the same cosine explains seasons — the Earth's 23.5° tilt makes summer sun strike North America near-vertically (concentrated) while winter sun arrives at a grazing angle (spread out, ~half the energy per area). [seasons figure]

• **units — radiometric vs photometric** (worth pinning down): the quantities above are **radiometric** (raw energy, in **watts**). Weight them by the eye's spectral sensitivity — the **luminous-efficiency curve $V(\lambda)$** — and you get the **photometric** quantities the photographic world actually quotes:

• radiant flux **W** ↔ luminous flux **lumen (lm)**

• irradiance **W/m²** ↔ **illuminance lux** (lm/m²) — light *arriving on* a surface (an incident-meter reading)

• radiance **W/m²/sr** ↔ **luminance cd/m²** (the "**nit**") — light *along a ray* / leaving a surface; this is the perceptual "brightness" of a patch, and what displays, HDR, and the photopic/scotopic scale (Perception) are all quoted in

• intensity **W/sr** ↔ luminous intensity **candela (cd)** — the SI **base** unit (≈ one candle)

• so **cd/m² = lm/(m²·sr)**; for scale, a monitor is ~100–300 cd/m², an HDR display ~1000+, paper in sunlight ~10⁴, and the photopic/scotopic ranges (Perception) live on this same axis

• **exposure** H = irradiance × time → ties straight back to aperture (D/f)² and shutter t (Exposure settings)

• **falloff**: a *point* source dims as **1/r²** (inverse-square), but scene **radiance is distance-invariant** — contrast the two carefully, it confuses people

• this radiometry underlies **HDR** (recovering scene radiance) and **computational illumination**

• (the off-axis **cos⁴** edge falloff / natural vignetting is deferred to **Sensors**, where the pixel irradiance integral and a finite aperture make it meaningful)

• light is a **wave**: at edges and small apertures it **diffracts** — bends and spreads instead of travelling in straight rays

• consequence for imaging: a **finite aperture cannot focus to a perfect point**; even an ideal lens images a point as a small blur, the **Airy disk**, whose size grows with wavelength × f-number

• the **diffraction limit**: the finest resolvable detail is set by wavelength and aperture (Rayleigh ≈ 1.22 λ/D); a **larger** aperture diffracts **less**

• so **stopping down too far softens** the image — aberrations dominate wide open, diffraction dominates stopped down → a sharpness **sweet spot** (developed with MTF in the Optics part)

• **coherent vs incoherent**: we mostly assume **incoherent** imaging (intensities add, stay positive, linear); coherent / wave optics returns for advanced optics (metalenses, lensless, holography)

• **a live demo — the wave equation, solved**: diffraction through a slit is shown as an actual 2-D wave simulation (FDTD), wavefronts bending and spreading past the opening, with the time-averaged intensity pattern alongside — and a companion widget where a *smaller aperture visibly spreads more* (θ ≈ λ/D). [`fig-diffraction-wave-sim` (animated), `fig-diffraction-aperture-size`]

• **the audio parallel** (use it — sound is the everyday 1-D wave): every wave effect here has a **sound** counterpart, and most people have *heard* them. Diffraction — **bass bends around a doorway** (long λ) while treble stays directional, exactly as a small aperture spreads light; plus **interference** and **beats**, **standing waves** in a pipe or room, and the **Doppler** shift. The governing math is shared (a wave equation, Fourier content, λ/D spreading), so acoustics is a reliable intuition pump for the optics — and the same FDTD solver in `fig-diffraction-wave-sim` *is* an acoustic simulation with light's numbers.

• side bar — **diffraction and duality, revisited with walkers**: diffraction is the *wave* story (a slit spreads light by θ ≈ λ/D); the *photon* story is that **individual** photons still arrive one at a time, yet pile up into the same fringe pattern. The **walking-droplet** analog (see [[#Light: rays, waves, and the spectrum]]) dramatizes the marriage: a single bouncing drop passing a slit is steered by its own **pilot wave** into a **diffraction-like** distribution — so you can literally *watch* a "particle" guided by a wave. [reuse the Couder/Bush walker video → [[Video Resources]]]

• light, lens, pupil, retina,

• cones/rods, distribution, fovea — note there are **no S (blue) cones in the very centre of the fovea**, and far more L+M than S overall, so **blue is low-resolution and a bit out of focus** (chromatic aberration) — yet we perceive blue everywhere (the brain fills it in). Luminance ≈ the **L+M cone sum**, which is why **green carries most of the acuity** (and why Bayer uses 2× green).

• cells, ganglions, on-retina spatial processing, visual cortex, V1, higher.

• **the eye as a camera — and its refractive errors.** Like any lens system the eye can mis-focus: **myopia** (eyeball too long / cornea too curved → image forms *in front* of the retina), **hyperopia** (too short → *behind*), **astigmatism** (a non-spherical cornea → two focal lines, not one point), and **presbyopia** (the aging lens stiffens and loses **accommodation**). Each is corrected by a lens quoted in **diopters** (D = 1/focal-length in metres): a negative/diverging lens for myopia, positive/converging for hyperopia, a **cylindrical** lens (with an axis) for astigmatism. [`fig-refractive-errors`]

• **how the prescription is *measured* (the machines), and why it's just inverse imaging.** **(1) Retinoscopy** — shine a moving streak of light into the eye and watch the **retinal reflex** sweep across the pupil; its direction and speed reveal myopia vs hyperopia, and trial lenses are stacked until the reflex stops moving ("neutralization"). **(2) Autorefractor** — automates this objectively: it projects an **infrared** target onto the retina and analyses the **returning image** (its focus/size, or a Hartmann–Shack / Scheimpflug pattern) to solve for the corrective lens that makes the retinal image sharp — essentially a little camera running autofocus *on your eye*, in seconds. **(3) Phoropter** — the familiar "better one… or two?" wheel of trial lenses over an eye chart: a **subjective** refinement of the machine's objective estimate. **(4) Wavefront aberrometer (Shack–Hartmann)** — for the full optical picture: a **lenslet array** samples the wavefront leaving the eye and measures **higher-order aberrations** (coma, spherical, trefoil), not just sphere/cylinder — the same wavefront sensing used in **adaptive optics** and in planning **LASIK**. The throughline: measuring an eye is **inverse imaging** — find the optics that would make the retinal image (or the returning wavefront) ideal. (Cross-ref wavefront / Shack–Hartmann sensing and adaptive optics in [[Optics, lenses, and aberration correction]].)

• cones, projection from spectrum to 3 responses

• relate to measurement equation in image formation

• **the response is a matrix–vector product**: discretize the spectrum into N wavelength bins → a vector E ∈ ℝᴺ; stack the three cone sensitivities as the **rows of a 3×N matrix C** (one row per L/M/S); the cone responses are then **r = C·E**, i.e. each r_k = Σ_i c_k(λ_i)·E(λ_i). In the real world wavelength is **continuous**, so it's really **infinite-dimensional** — the sum becomes the integral r_k = ∫ c_k(λ)·E(λ) dλ. [cone-response matrix figure]

• insist a given cone does not make the difference between wavelength. Example of monochromatic stimuli

• point out they are non orthogonal and lots of stuff can’t be negative.

• This make linear algebra more messy

• 💡 **Big lesson:** **color is non-orthogonal *and* non-negative** — overlapping cone axes plus the no-negative-light constraint are what make color algebra tricky (analysis needs a dual basis with negative coordinates).

• side bar: **non-orthogonal bases & the dual basis** — the cone "axes" are **not orthogonal** (L and M overlap heavily), so the comfortable orthonormal intuition fails. Picture two basis vectors **c₁, c₂ both in the positive quadrant** of the (canonical) spectrum space, each standing for one cone. **Synthesis** is easy — any point is a combination a₁c₁ + a₂c₂ (a parallelogram). But **analysis** — recovering the coordinates a₁, a₂ — is **not** a plain projection onto c₁, c₂; you must project onto the **dual (reciprocal) basis** c₁\*, c₂\* (defined by cᵢ\*·cⱼ = δᵢⱼ). For a non-orthogonal positive basis the **dual vectors point partly into the negative quadrants** — they have **negative coordinates** in the original space. That is exactly why "stuff can't be negative" makes color algebra messy: the natural *analysis* vectors are negative. (Only for an **orthonormal** basis do a basis and its dual coincide — then analysis = synthesis = projection.) [non-orthogonal dual-basis figure]

• color blindness. Linear algebra perspective (projection loses info)

• we are all color blind

• side bar — **animal vision and the divergence of opsins**: "color" is just whatever set of **opsins** (photopigments) an animal carries, and these have **diverged enormously** across major animal groups — many insects are UV–blue–green trichromats, birds and reptiles are often **tetrachromats** (a fourth, UV cone), most mammals reverted to **dichromacy** (primate trichromacy is *re-evolved*, via a gene duplication), and the **mantis shrimp** carries ~12–16 photoreceptor classes. So human red–green color blindness is one point in a vast space — every species is "color blind" relative to some other. The opsin family tree and its divergence are traced in the "Evolution of Eyes" chapter of *Vision* (Cambridge); the schematic opsin gene tree (ancestral opsin → rod RH1 + cone classes, human S/M/L with the recent L/M duplication) is [fig-opsin-tree]. [→ Animal eyes]

• the **multistage color model** — the brain re-codes color in stages (simplified; see Reinhard et al., *Color Imaging* for the full version):

• **stage 1 — cones**: three heavily-overlapping LMS responses (above)

• **stage 2 — opponent recoding** in the retina / LGN: LMS → **light–dark, red–green, blue–yellow** (decorrelates the channels, matches the four unique hues; → next section)

• **stage 3 — appearance level**: a higher organization closer to **hue / saturation / brightness** (an HSV-like arrangement) where context, adaptation and memory act (→ color appearance models, Color technology)

• **focal colors & naming**: color *names* cluster around shared **focal colors** across languages (Berlin & Kay; the World Color Survey) — evidence the recoding is perceptual, not arbitrary

• experimental support: **psychophysics** (hue-cancellation — Hurvich & Jameson) and **physiology** (opponent cells — De Valois)

• the perennial muddle (RGB? RYB? CMY?) has *two separate* causes, neither of which is a fact about light:

• **dual-process / opponent coding** (next section): the visual system re-wires the three cone signals into **red–green, blue–yellow, light–dark**, so *red, green, blue, and yellow* all feel "primary" perceptually — there are **four** psychological unique hues, not three. This is why RYB feels natural to painters and why the cones' actual L/M/S don't match anyone's intuition of "primaries."

• the rest is **additive vs subtractive synthesis** (Color technology): **RGB** are the *additive* (light) primaries, **CMY** the *subtractive* (ink/filter) primaries; grade-school **RYB** is a folk version of subtractive mixing. So "primary" conflates a **perception** fact with a **technology** choice — different goals, different "primaries."

• source: lecture — the opponent rewiring "is one of the many causes of all the confusion about what's a primary color" (transcript 10/2); pair with [[Glossary]] entry for *primary*.

• two stages: trichromatic at the cones, then opponent in the retina / LGN — red–green, blue–yellow, light–dark

• explains afterimages and color naming; and why **luma/chroma** (YCbCr — luma Y′ on gamma-encoded RGB, see *Linear vs Gamma*) compresses well (see JPEG)

• we recenter the neural response around the current average brightness — spanning a huge range over *time*, not all at once

• mechanisms: neural, chemical (photopigment bleaching), and the pupil

• afterimages; the Spanish-Castle illusion

• **the operating regimes — photopic, mesopic, scotopic** (which receptors are working, and *how bright* that is, in **luminance** cd/m²):

• **photopic** (daylight, **cones**, full color, sharp foveal vision): roughly **> 3 cd/m²** — office light ~10²; an overcast sky ~10³–10⁴; a sunlit scene ~10⁴–10⁵

• **mesopic** (dusk / dim interior, **rods *and* cones together**): roughly **10⁻³ – 3 cd/m²** — colors desaturate and the **Purkinje shift** sets in (as rods take over, blues look relatively brighter and reds darker)

• **scotopic** (night, **rods only** — no color, poor acuity, and *off-fovea* since the fovea is all cones): below ~**10⁻³ cd/m²** — moonlit landscape ~10⁻²; starlight ~10⁻³–10⁻⁴; the **absolute threshold** of vision ~**10⁻⁶ cd/m²**

• big picture: the eye works across a staggering **~10⁻⁶ to ~10⁸ cd/m²** (≈ 14 log units), but only over a few log units **at once** — which is exactly why **adaptation** (above) and, in cameras, **HDR / tone mapping** exist. [fig-photopic-scotopic-range]

• the visual system reports a surface's intrinsic **lightness** (its reflectance / albedo), **not** the raw luminance reaching the eye — a white page in shadow still looks white, a black one in sunlight still looks black, even when the shadowed white sends *less* light to the eye than the sunlit black

• it **discounts the illuminant**: the brain factors the retinal image into **illumination × reflectance** (the same entanglement from Light and physics) and reports the *reflectance* — the property we actually care about

• this is an **ill-posed inversion** (one image, two unknowns), so the brain leans on cues and priors: **edges and context** tell it where the *illumination* changes versus where the *surface* changes

• the canonical demo: **Adelson's checker-shadow illusion** — two squares of identical luminance look very different because the brain infers the cast shadow and discounts it

• **Retinex** (Land & McCann) models the same idea computationally: treat slow spatial gradients as illumination and sharp edges as reflectance changes, then integrate the edges back to lightness

• ties to **intrinsic images** (split a photo into reflectance + shading) and to the darkroom craft of **dodging & burning** / tone mapping

• the **color analog** of lightness constancy: we perceive an object's intrinsic **surface color** (its reflectance) despite large changes in the **illuminant** — a white shirt reads white under noon daylight, a warm lamp, or blue shade, even though the spectra reaching the eye differ a lot

• again the brain **discounts the illuminant**: it estimates the light's color and divides it out so the surface reads consistently — the same ill-posed (**illuminant × reflectance**) inversion, solved with assumptions (e.g. the average scene is roughly gray, or the brightest patch is white)

• a key mechanism is **von Kries adaptation**: an *independent gain on each cone type* (L, M, S) rescales the response to the prevailing light — plus spatial / contextual cues; it is **not perfect** (some cast remains, and memory colors bias us)

• this is the **biological counterpart of white balance** (Color technology) — a camera's automatic white balance must do, by algorithm, what our visual system does for free

• classics: **Retinex** (Land & McCann); **Bayesian** color constancy (Freeman & Brainard); gray-world / bright-pixel heuristics

• the **"the dress"** demo (2015): the *same* image pixels look **blue-black** to some viewers and **white-gold** to others — because the brain must *guess the illuminant* (cool daylight vs warm indoor light) and divides it out differently. A perfect, viral illustration that color constancy is an **ambiguous inference**, not a measurement — the visual system commits to a prior and different people commit differently. Ties to white balance (a camera faces the exact same ill-posed guess).

• contrast is about *ratios*: 1→2 looks like 100→200 (discounts the multiplicative effect of light) — foreshadows gamma / log encoding

• Weber's law / just-noticeable difference

• simultaneous contrast; the Adelson checker-shadow illusion

• Visual Acuity

• Simultaneous contrast

• Mach bands; lateral inhibition (center-surround receptive fields)

• **CSF** — measured by **psychophysics** (show **sine-wave gratings**, ask "do you see a grating?"); the **contrast sensitivity function** is band-pass in spatial frequency. Crucially the **chromatic** CSFs (red–green, blue–yellow) cut off at **far lower** spatial frequency than the **luminance** CSF — we resolve **color** much more coarsely than **brightness**. [three-CSF figure]

• the dramatic demo: in Lab, **blur the chroma** channels and the image still looks sharp; blur the **luminance** and it falls apart — the empirical basis for **chroma subsampling** (JPEG / YCbCr) and a confirmation of the opponent model. [blur-chroma vs blur-luminance figure]

• the visual system trades some spatial resolution for **temporal** resolution: a **temporal CSF** band-passes flicker, with a **critical flicker-fusion** frequency (~50–60 Hz in bright light, lower when dim) above which a flickering source looks steady — why displays/film refresh fast enough to fuse, and why old CRTs / fluorescents could flicker in peripheral vision

• **apparent motion** (the phi phenomenon) + persistence of vision make a sequence of frames read as continuous movement (→ Motion/Video)

• **temporal adaptation** (light/dark adaptation over seconds–minutes; → Light adaptation) and the eye's own motion (saccades, microsaccades) shape what we see over time

• exploited by computation: **frame-rate** choices, flicker-free capture, and **video magnification** (amplifying tiny temporal changes — Motion/Video)

• attention; fixations; saccades; smooth pursuit

• the eye samples a scene through rapid saccades + a high-res fovea

• saliency (what draws the eye) is modeled computationally → see [[Human factors#Computational models of perception]] (the model and its uses) — the ML estimator is in [[Computational tools#Machine learning]]

• color reproduction kind of stuff

• blue shift

• colorfulness as a function of luminance (Hunt effect)

• hue shifts: Bezold–Brücke (with intensity), Abney effect (with added white)

• analysis = *sensing* color (project a spectrum onto 3 cone / sensor responses)

• synthesis = *reproducing* color (add a few primaries)

• both lose information: the projection is non-orthogonal and light can't be negative — a tension that recurs (sensing, displays, white balance)

• the color-matching experiment: match a test light by mixing 3 primaries

• trichromatic theory (von Helmholtz 1859); tristimulus values

• metamers: different spectra that match — *why* color reproduction is possible at all

• CIE 1931: color-matching functions, the xy chromaticity diagram, the spectral locus, the white point

• negative matches → why some colors need "impossible" primaries (see Reproducing color)

• different challenges: quantization, noise

• same observation: ratios matter

• **side bar — why gamma ≈ 2.2? the CRT accident**: old **CRT** displays had a *natural* power-law response — screen luminance ∝ (grid voltage)^γ, γ ≈ 2.2–2.5 (electron-gun physics: beam current vs control-grid voltage is a power law). So a CRT **automatically decodes** a gamma-encoded signal: send V = L^(1/γ) and the tube returns L = V^γ (linear light), no decode hardware. **sRGB's ≈2.2 codifies it.** Bonus: that curve ≈ the *inverse of human lightness perception* (Weber–Fechner), so it also puts code levels where the eye is sensitive — **why we kept gamma after CRTs vanished** (LCD/OLED emulate the curve for compatibility). [CRT-response figure]

• **side bar — even sRGB isn't quite continuous**: the ubiquitous sRGB transfer curve is **piecewise** — a short **linear** toe near black, $V = 12.92\,L$ for $L \le 0.0031308$, then a **power** segment $V = 1.055\,L^{1/2.4} - 0.055$ above it — meant to join smoothly at the knee. But the standardized constants (the $12.92$ slope, the $0.0031308$ breakpoint, the $0.055$ offset, the $1/2.4$ exponent) are **rounded and not mutually consistent**, so the two pieces **don't exactly meet**: at the breakpoint the linear part gives $\approx 0.04045$ while the power part gives $\approx 0.04059$ — a tiny **discontinuity** ($\sim 10^{-4}$), on top of the slope kink that's there by design. So even the encoding *every* image uses is, strictly, **not continuous** at the join. The lesson: real transfer functions are messier than the clean "$\gamma \approx 2.2$" story — **check the actual curve** (and its inverse) before doing arithmetic or quantizing in that space. *(A "corrected" sRGB nudges the breakpoint/scale so the pieces meet exactly.)*

• **side bar — gamma in film**: the word **gamma** comes from photography. Film's response is the **characteristic (Hurter–Driffield) curve** — **density** vs **log exposure** — with a toe (shadows), straight middle, and shoulder (highlights). Film **gamma = the slope of the straight portion** (γ = ΔD/Δlog H) = the medium's **contrast**, set by development (push-processing raises γ). So *film* gamma is a contrast/rendering parameter while *display* gamma is an encoding power law — both log–log slopes, different jobs. End-to-end **system gamma** (capture × display) is tuned slightly >1 (~1.1–1.5) so images look right in a **dim viewing surround**. [film characteristic-curve figure]

• 💡 **Big lesson:** **additive vs multiplicative → choice of encoding.** Light *adds* (incoherent sources, blur) but surfaces and contrast *multiply*; **log** makes the multiplicative native, **linear** suits the additive (deconvolution), **gamma** is the compromise with least crazy behaviour near zero.

• **log is becoming the standard for *video***: high-end and now prosumer cameras capture in **log** profiles — **S-Log / S-Log3** (Sony), **Log-C** (ARRI), **V-Log** (Panasonic), **C-Log** (Canon), **RED Log3G10** — and the **ACES** pipeline standardises a log working space (ACEScc / ACEScct). Log packs a wide scene dynamic range into limited bits while keeping the grade malleable: a **stop** becomes a roughly constant code increment, so exposure and contrast moves are uniform. The footage looks flat and desaturated straight out of camera and is **color-graded** to a display encoding; the broadcast-HDR transfer functions **HLG** and **PQ** (SMPTE ST 2084) are the display-side cousins. So in video the trio increasingly resolves to **log for capture, gamma / PQ / HLG for display**. (→ HDR, tone mapping.)

• **luma vs luminance** (a recurring confusion): **luminance** Y is the *linear*, perceptual brightness — a weighted sum of **linear** RGB (CIE Y); **luma** Y′ is the *same-shaped* weighted sum computed on the **gamma-encoded** RGB (the coding convenience behind YUV / YCbCr / JPEG). They differ because the weighting and the gamma don't commute.

• **our convention**: use **Rec. 601 weights — Y = 0.299 R + 0.587 G + 0.114 B** — matching video / JPEG / YCbCr and our programming exercises; mention **Rec. 709** (0.2126 / 0.7152 / 0.0722) as the HDTV / linear-light alternative.

• **RGB and its flavours**: many RGB spaces differ only by their **primaries** (gamut) and **white point**, related by a single **3×3 matrix M** — sRGB / Rec. 709 (web, HDTV), **Adobe RGB** and **Display P3** (wider), **Rec. 2020** (UHD), **ProPhoto** (very wide, for editing). Converting between them (or to XYZ) is one matrix multiply *in linear light*.

• **opponent / luma-chroma spaces** (YCbCr, YUV, YCoCg): a fixed 3×3 map separating **luma** from two **chroma** axes — used for compression (chroma subsampling) and some editing.

• caveat: "linear" here means **a linear map between spaces**, not necessarily linear-*light* values — sRGB carries a gamma; XYZ is linear-light.

• **CIELAB** (and CIELUV): a deliberately **non-linear, perceptually-uniform** space — a cube-root compression of XYZ about a white point yields **L\*** (lightness) and **a\*, b\*** (opponent chroma), so **equal distances ≈ equal perceived differences**; color difference **ΔE = ‖ΔLab‖** (improved by ΔE2000)

• the motivating demo: sample a rainbow at **N colors equally spaced in sRGB vs in CIELAB** — the sRGB steps have wildly uneven perceived gaps (ΔE), the Lab steps are even [perceptual-uniformity figure]

• **why non-linear**: perception is roughly power-law (Weber–Fechner / Stevens), so a uniform space must warp XYZ — the same reason gamma encoding exists

• **friends**: HSV / HSL (handy for UI, *not* perceptually uniform), and modern uniform spaces — CAM02-UCS, **Oklab** — that fix CIELAB's known non-uniformities (the *Chong* perceptual color metric belongs here)

• intro: fundamental challenge of spectrum non-orthogonality.

• Give example of reproducing a monochromatic signal.

• Show 2D linear algebra equivalent problem.

• Say that the big extra challenge is that we can only have positive light.

• Non-orthogonality + restriction to positive -> impossible problems.

• **additive vs subtractive** color synthesis — cartoon: split the spectrum into 3 equal bands and pretend they are pure **B, G, R** [3-band synthesis figure]

• **additive** (lights): start from black and *add* bands → R+G+B = white; pairs give yellow / cyan / magenta

• **subtractive** (filters/inks): start from white and each filter *removes* a band — but this is really a wavelength-by-wavelength **multiplication** of the spectrum, so "subtractive" is **misleading**; C·M·Y → black

• concrete proof it's a multiplication: a **yellow** filter (blocks blue) stacked with a **cyan** filter (blocks red) passes only the overlapping green band → **yellow × cyan = green**, not a "sum" of the two [subtractive spectra-multiplication figure: T_Y(λ)·T_C(λ)]

• the **overlapping-disks** picture makes the same point at a glance: RGB lights overlapping toward white (additive) vs CMY inks overlapping toward black (subtractive) [overlapping RGB/CMY disks figure]

• real systems are often **hybrid** — some light *added* and some *removed* — e.g. an **LCD projector**: a white lamp, LCD panels that *subtract* (modulate) each channel, then the three channels *added* on the screen

• various technology: CRT, LCD, projectors (DMD, LCD, laser)

• film, printer, halftoning

• gamut, gamut mapping

• **saturation**: scale the chroma — push colors away from the grey (luma) axis in an opponent / HSV space; a uniform scale over-saturates already-vivid colors and skin

• **vibrance**: a *smarter* saturation — boost **less-saturated** colors more and **protect** already-saturated ones and **skin tones** (a non-linear, hue-aware gain)

• **selective / HSL color** (Lightroom's mixer): per-hue-band hue/saturation/luminance, and **color grading** (shadows / mids / highlights tinting) from film & video

• the *mechanics* (curves, LUTs, pipeline placement) live in **Basic Image Processing** (point operations); here we own the **color-space** reasoning — which axis, which space, why protect skin

• the key framing: "convert to black & white" is **under-specified** — you are projecting **three numbers to one**, an inherently **lossy, many-to-one** map, so there are *many* valid answers and the right one depends on **intent**, not correctness

• **pick a single channel** (R, G, or B): quick; occasionally useful (the **green** channel is close to luminance and usually least noisy; the **red** channel smooths skin and darkens skies) — but it throws away two-thirds of the data

• **average** $(R+G+B)/3$: simplest, but **perceptually wrong** — it makes blue as bright as green; skies and foliage come out muddy

• **weighted luminance** (the perceptual default): a weighted sum that matches the eye's sensitivity — **green dominates**, blue counts little — **Rec. 601** $0.299/0.587/0.114$ or **Rec. 709** $0.2126/0.7152/0.0722$; this is exactly the cone-weighting / luminance idea from [[Perceptual color and trichromatic vision]]

• **but in which space?** the recurring **[[#Linear vs gamma vs. log encoding|luma vs luminance]]** distinction — the cheap version weights the **gamma-encoded** values (**luma** Y′, what most software does), the radiometrically correct version weights **linear** RGB (**luminance** Y) then re-encodes; they give visibly different greys (linear-light is darker in saturated reds/blues)

• **lightness, not luminance**: take **L\*** from **CIELAB** (perceptually uniform lightness) — or the UI-space shortcuts **HSL "lightness"** $(\max+\min)/2$ and **HSV "value"** $\max(R,G,B)$ — each a different grey again

• **custom channel mixer** (the artistic control): choose the weights yourself (keep $\sum w_i\approx 1$ to hold exposure) — this is Lightroom/Photoshop's **B&W mixer**, and the digital descendant of the darkroom **color filter on B&W film**: a **red** filter darkens a blue sky for dramatic clouds, a **yellow/green** filter lightens foliage, an **orange** filter smooths skin (Ansel Adams's black skies are a red filter on panchromatic film). So a B&W conversion is really a decision about *how each color should map to grey*

• **the hard case — isoluminant colors collapse**: two distinct hues with the *same* luminance (a red and a green of equal brightness) map to the **same grey**, erasing the edge between them; **contrast-preserving decolorization** methods (Color2Gray, Gooch et al. 2005; Grundland & Dodgson 2007) optimise the mapping to keep **color contrast** rather than only luminance — the principled answer to "the green apple and red apple shouldn't vanish into one grey"

• dimensions of color perception:hue, chroma, etc.

• perceptual color space

• HSV and other UI-spaces

• color appearance models

• a *memory color*: viewers are unusually sensitive to skin looking right

• the skin-tone locus; cameras and film are tuned to render skin pleasingly

• diversity of skin tones and historical bias (e.g. Shirley cards) — a fairness tie-in (see ethics)

• pinhole,

• show camera obscure example (some artist)

• **geometry of pinhole imaging**: each scene point's chief ray passes through the pinhole onto the sensor. The physical sensor sits *behind* the pinhole, so the image is **inverted** (upside-down and left-right flipped).

• we instead use a **virtual image plane the same distance in front** of the pinhole — *not physical*, but it keeps the image **right-side up**, so we adopt it throughout (consistent with the camera looking down **+z** → [[Conventions, Notation & Style#Geometry & coordinate systems]]).

• the pinhole→sensor distance is the **focal length f**: a short f gives a **wide** field of view, a long f a **narrow** (telephoto) one — FOV = 2·arctan(sensor / 2f).

• **side bar — DIY pinhole.** A pinhole needs no lens, so it is the easiest real camera to build: punch a clean tiny hole in foil and tape it over the lens mount of a camera body, or over one end of a lightproof box (a shoebox / oatmeal tin) with film or paper at the other — a working **camera obscura**. Smaller holes are sharper but dimmer (diffraction sets the floor), so exposures run long.

• **side bar — the pinhole bag** (credit the idea to **Freeman, Torralba & Isola**'s online vision book, their Figure 5.5, at <https://visionbook.mit.edu/imaging.html>): pull a bag with a single small pinhole over your head, let your eyes dark-adapt, and you see the **upside-down image of the bright world outside projected on the inside of the bag** — pinhole projection made personal, and an unforgettable demo that the inverted image is real.

• exercise: make your own, do in a room

• mention animal eyes

• linear perspective when camera is at origin and along z axis

• **perspective projection equation**: a scene point (X, Y, Z) projects to screen coordinates **x' = f·X/Z** and **y' = f·Y/Z** — each screen coordinate is the world coordinate **divided by the depth Z and multiplied by the focal length f** (the "divide by z" that makes distant things small). [shown both as 2D similar-triangles and in a 3D view]

• 💡 **Big lesson — with the camera at the origin looking down +z, perspective is *mostly a division by depth*.** In that canonical frame the whole of linear (pinhole) perspective is just $x' = f\,X/Z,\ y' = f\,Y/Z$: divide each world coordinate by the depth $Z$, then scale by $f$. Everything else a camera does — sitting somewhere else, pointing some other way, landing on pixels — is bolted on *around* this one step. The **divide-by-depth is the whole of perspective**; the rest is bookkeeping. → register [[Big Lessons]]

• **homogeneous coordinates**: write a 2D point as (x, y, 1) up to scale, so projection, **translation**, and camera motion all become **matrix multiplies**

• **translation becomes a matrix, too.** A shift by $(t_x, t_y)$ is *not* linear on $(x, y)$, but on the homogeneous triple it is an ordinary matrix multiply: $\begin{bmatrix}1&0&t_x\\0&1&t_y\\0&0&1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}x+t_x\\y+t_y\\1\end{bmatrix}$, and in 3D the $4\times4$ analogue $\begin{bmatrix}I_3&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$ translates $(X, Y, Z, 1)$. This is *the* reason graphics and vision live in homogeneous coordinates — translation, rotation, projection, and the perspective divide all compose into one chain of matrix multiplies. [translation-matrix in homogeneous coordinates]

• so far the camera has sat at the **origin looking down +z**, where perspective is the clean divide-by-depth above. But a real camera is **somewhere else, pointing some other way** — so before we can use that canonical projection we must re-express the scene **in the camera's own frame**. This is the step that turns "an arbitrary camera" into "the camera at the origin."

• 💡 **moving the camera = applying the *inverse* transform to the world.** Sliding the camera right by $d$ is indistinguishable from sliding the whole **world left** by $d$; rotating the camera one way is rotating the world the other. So we never move the camera *inside* the projection — we move the **world into the camera's frame** by the inverse rigid transform, then project with the fixed origin/+z pinhole. (This is also why "camera-to-world" and "world-to-camera" matrices are inverses — the convention trap below.)

• that world→camera step is a **rotation R then a translation t**, written as one $4\times4$ matrix in homogeneous coordinates, $\begin{bmatrix}R&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$, carrying a world point $(X, Y, Z, 1)$ into the camera frame — where the canonical divide-by-depth then applies. Translating the camera is changing $\mathbf{t}$; aiming it is changing $R$.

• 💡 **Big lesson — perspective for an *arbitrary* camera is one $4\times4$ matrix followed by a divide by the last coordinate.** Pose the camera (world→camera $4\times4$), project, and map to pixels — every stage is a matrix on the homogeneous point, so the *whole* geometric pipeline composes into **one matrix multiply**, and the only nonlinearity — the perspective divide — *is* the final "divide through by the last coordinate." The arbitrary-camera case is the origin case (divide by depth) wrapped in two more matrices; homogeneous coordinates are exactly what make that wrapping elegant. → register [[Big Lessons]]

• moving camera, matrices, all that good stuff

• **camera intrinsics K** — give them a real treatment, not a bullet: reason first in **normalized image coordinates** (project onto the z=1 plane, independent of sensor size), then map to **pixels** with a second, **affine** matrix carrying the focal lengths fx, fy and the **principal point** (the arbitrary image centre). So the full projection **decomposes** as *(pixel affine) · (project at z=1) · [R|t]* = K·[R|t]. This split is also why **cropping an image = using a longer focal length** — you change only the pixel mapping, not the projection. [projection-matrix decomposition figure; crop = focal-length figure]

• 💡 **Big lesson — cropping a photo *is* changing the focal length.** From the projection geometry, cropping touches only the last stage, the pixel mapping $K$: keep a smaller patch of the normalized image and rescale it to fill the frame — exactly what enlarging $f_x, f_y$ does. A 2× crop and a 2× zoom produce the *same* picture (the crop just has fewer pixels). Scaled up to whole sensors, this same fact is the **crop factor** that relates sensor sizes. → register [[Big Lessons]]

• **camera extrinsics [R|t]** — where intrinsics are *inside* the camera, extrinsics are the camera's **pose in the world**: a **rigid 6-DOF** world→camera transform, a rotation **R** (3×3, 3 DOF — orientation) plus a translation **t** (3 DOF — position). Together they place the world into the camera's frame *before* projection, so the **full projection is intrinsics · extrinsics = K·[R|t]** (the [R|t] moves the world into camera coordinates; K projects and maps to pixels). "Moving the camera" is just changing R and t.

• **the convention trap — cam2world vs. world2cam** (the one place to slow down): the matrix [R|t] above maps **world → camera**; its inverse maps **camera → world**. Worse, packages disagree on whether **t is the camera's position** (its center C in world coordinates) or its **negation** in the projection (t = −R·C). Mixing the two silently flips signs / mirrors a reconstruction — always pin down *which direction* a stored pose maps and *what t means* before trusting it.

• this is exactly the machinery that **multiple-view geometry** and **panorama homographies** rely on — relating two cameras is composing their extrinsics; a pure rotation between views collapses [R|t] into a **homography** (→ [[Multiple view geometry]]; Pano & homographies).

• vanishing points

• **what projection preserves vs destroys**: straight **lines → lines** and incidence survive; **angles and lengths do not** (parallels converge to vanishing points, a circle can become an ellipse) — the core fact behind perspective

• Photography, focal length, field of view, various trigomometric ideas, magnification. Difference focal length vs distance and how that changes the background

• the same focal-length-vs-distance distinction governs **portraiture**: facial proportions (and even perceived personality) are set by **camera-to-subject distance**, which is *why* portrait lenses (~85–135 mm) flatter — they let you back up (Perona 2007; full treatment in [[#Lenses and focal length]]).

• sensor size and impact on perspective. Maybe too late?

• sidebar: the (weird) meaning of **"35 mm"** = a **24×36 mm** image frame. The "35 mm" is the *width of the film stock* (including the sprocket-hole margins), not the image; Barnack ran 35 mm cine film sideways, giving a 36 mm × 24 mm frame. Today this size is called **"full frame"**, and smaller sensors are quoted by their **crop factor** relative to it (APS-C ≈ 1.5×, Micro 4/3 ≈ 2×, phone sensors much smaller).

• **depth vs. ray length**: a depth sensor's "depth" is the **z coordinate**, *not* the ray length ‖·‖ to the point — keep them distinct; **unproject** (pixel + depth → 3D point) inverts the projection (RealSense & friends do this live). Depth sensors exist (→ ToF / structured light / stereo). [depth-vs-ray-length / unproject figure]

• a real lens is a piece of glass bounded by **two curved surfaces** (two interfaces)

• light bends at each interface by **refraction** — Snell's law: n₁ sin θ₁ = n₂ sin θ₂

• shown explicitly on a **thick lens** (the two interfaces drawn far apart): the angle of incidence and the angle of refraction at *each* surface, with the indices n₁ (air) and n₂ (glass) marked. [thick-lens Snell figure]

• a curved interface bends a bundle of rays; the two surfaces of a single lens together steer parallel light toward a **focal point**

• **a second view — Fermat / equal optical path length**: a lens focuses because **every** ray from the object to the focus has the *same optical path length* — the longer geometric detour through the thick centre is exactly compensated by the slower speed (higher index) of glass there, so all paths arrive **in phase** and interfere constructively. The ray (Snell) picture and this wave (Fermat) picture are two takes on the one fact. [equal-path-length figure]

• 🖱️ **Interactive (web edition):** a live **2-D wave simulation of a thick lens** — a point source on the left radiates circular waves that cross a biconvex glass slab (slower inside, higher index n); because the glass is thickest on the axis it retards the centre of each wavefront most, turning the diverging wave into a **converging** one that collapses to an **image point**. This is the Fermat/wave picture made literal: focusing emerges from the wave equation, not from a ray rule. Sliders for the surface curvature R, the index n, and the source distance u, with the focus tracking the **lensmaker's equation** $1/f=(n-1)\,2/R$ and the **imaging equation** $1/v=1/f-1/u$ live on the figure. The lens sits in an opaque mount (aperture stop) so all light is forced through the glass; absorbing (sponge) borders, so nothing reflects off the window edges. Companion to `fig-refraction-wave-sim` (single interface) and `fig-diffraction-wave-sim`. [`fig-thick-lens-wave-sim`; interactive, FDTD]

• **lensmaker's equation** ties focal length to the two radii of curvature and the glass index: 1/f = (n−1)(1/R₁ − 1/R₂)

• this already assumes small angles (paraxial); at large angles the rays do **not** meet at one point → **aberrations** (forward ref: Optics part, where compound lenses correct them)

• eyes have optics too (cornea + lens are the two main refracting elements)

• the **thin lens** as a *simplification / linearization* of the single lens: collapse the two interfaces into one plane, assume small angles and negligible thickness → all bending happens once, at the lens plane (forward ref: Optics part for the accurate thick / compound model)

• precisely, **paraxial = first-order optics**: replace **sin θ ≈ θ** (keep only the linear term). Keeping the *next* term — sin θ ≈ θ − θ³/6 (**third-order optics**) — is exactly what produces the **Seidel aberrations**, so "linearization" and "no aberrations" are the *same* approximation (forward ref: Optics part)

• three-ray construction (parallel→focus, through-centre, focus→parallel) locates the image

• **focus equation** 1/f = 1/u + 1/v (object distance u, image distance v); an object at infinity focuses at f; **focusing** = changing v (move the lens or the sensor) to image nearer/farther objects

• **conjugates**: each object distance pairs with one image distance — object at ∞ → image at f; far → just beyond f; close → far behind; at the focal point F → rays exit parallel (image at ∞). [conjugates figure]

• **symmetry & reversibility**: the focus equation $1/f = 1/s_o + 1/s_i$ is **symmetric** in object and image distance, so object and image are **conjugate** — **swap them and the rays retrace the same path** (the reversibility of light paths). Object and image planes are **conjugate planes**, and the magnification follows directly ($m = -s_i/s_o$, so swapping inverts it). This object↔image **reciprocity** recurs throughout optics (e.g. why a lens images either way, and why entrance/exit pupils are conjugates).

• **magnification** m = −v/u (the image is inverted)

• **lensmaker's equation** (thin-lens form) 1/f = (n−1)(1/R₁ − 1/R₂) — connects f to the glass shape (from the single-lens section)

• **f-number** N = f/D sets the aperture (→ exposure, → blur)

• (depth of field gets its **own chapter**, next)

• (placed here, just before **sensors**, because it pays off as the framing for what a pixel actually measures)

• the **plenoptic function** describes *all the light filling space*: the radiance of every **ray**, parameterized by its 3D position, 2D direction, wavelength, and time — a 7-dimensional function [Adelson & Bergen]

• **any** imaging operation is the **measurement of some portion** of this function; the eye and the camera each sample a slice of it

• a conventional camera records only a **2D projection**: each pixel integrates all the rays reaching it, **discarding their direction** — and that discarded *directional* radiance (the plenoptic function restricted to free space, reduced to **4-D**) is exactly the **light field**. Record it instead of integrating it away and you can choose focus, aperture, and viewpoint *after* the shutter — introduced here, developed in [[14 Light fields and plenoptic cameras]].

• a recurring unifying view: imaging = sampling the plenoptic function

• **the chapter's thesis in one equation — a pixel value is an integral of the [[#The plenoptic function|plenoptic function]]** against a sampling kernel: each photosite *counts photons*, so it reports the **incoming radiance integrated over four physical dimensions**, weighted by the pixel's response. Schematically,

• pixel value ≈ ∫_time ∫_λ ∫_aperture ∫_(pixel area) L(x, y, θ, φ, λ, t) · r(λ) · (geometry) dA dω dλ dt

• i.e. **measurement = integrate the plenoptic function against the pixel's sampling kernel** — and the kernel's *support* in each dimension is exactly a photographic knob. [fig-pixel-integral: thin lens as an ellipse (aperture solid angle), the sensor, one finite pixel; the integral over aperture + pixel area]

• **each integration dimension ↔ a photographic parameter ↔ a downstream effect** (every later sensing topic is "what happens when you change the support of one axis"):

• **sensor footprint (x, y area)** — the pixel integrates radiance over its **finite area** on the sensor, which sets **resolution** and makes a sample a *box-filtered* average rather than a point sample → **sampling & anti-aliasing** (a bigger footprint blurs but suppresses aliasing; pre-filter vs. point-sample lives here).

• **aperture (solid angle of admitted rays)** — the pixel integrates over the **cone of directions** the lens admits, set by the **f-number** N = f/D. Wider aperture = more light *and* a wider integration cone = a shallower **depth of field** (out-of-focus points spread their energy across many pixels → the circle of confusion). The light-vs-blur trade is this single axis. (→ [[#Depth of field]])

• **exposure time** — the pixel integrates over the **shutter interval** t, which sets exposure (H = E·t) and, when the scene or camera moves during t, produces **motion blur** (the width of the temporal kernel). Light-vs-sharpness, the temporal sibling of aperture. (→ Motion / video)

• **wavelength × spectral sensitivity** — the pixel integrates radiance against its **spectral response** r(λ), collapsing a whole spectrum to one number (exactly like a cone, → [[#Color]]). A **color-filter array** gives each photosite a different r(λ) (R/G/B), so color is *spatially multiplexed* and must be reconstructed → **demosaicking** (Bayer).

• **why this framing pays off**: the rest of the sensing story is just *which kernel, how wide, and what it loses* — antialiasing widens the spatial kernel, depth of field is the aperture kernel, motion blur is the time kernel, demosaicking inverts the wavelength multiplexing. Each is also a **source of noise**: integrating fewer photons (small footprint, short time, narrow aperture) means a noisier estimate (→ Noise). Capture is **linear** in radiance, which is why this integral is well-defined and why **raw** values *mean* something physical.

• the **digital-sensor revolution came in two steps**: first **electronic** — turning light into an *analog electronic signal* (the photosite + readout; the era of the CCD and analog electronic / video imaging) — and only **then digital** — putting an **analog-to-digital converter (ADC)** on the path so that signal becomes **numbers** we can compute on. Computational photography lives on that second step; flag the distinction up front.

• a **sensor** is a 2-D array of **photosites** (pixels), each a potential well that **counts photons** → charge → voltage → a digital number; counting photons is what makes capture ~**linear** in radiance (keep this short — the deep sensor zoo is Advanced)

• a **microlens** over each photosite gathers light onto its active area (**fill factor** < 100% because of wiring), and a **color filter** sits under it → the **Bayer** mosaic (forward-ref: Demosaicking). [sensor cross-section figure]

• the scene-radiance → pixel-value mapping is the **camera response curve**: **raw** is ~linear in radiance, while the rendered (JPEG) image applies a non-linear response (≈ gamma + a tone curve — see Linear-vs-gamma in Color technology); **recovering** that response is exactly what HDR merging needs

• **silicon sees IR**: the band gap sits near **1100 nm**, so a photosite keeps counting well into the **near-infrared** — which is exactly why every color camera cements an **infrared-cut ("hot mirror")** filter in front of the sensor (the long-wavelength tails on the spectral-sensitivity curves are trimmed by *that filter*, not by the silicon).

• **make an IR camera**: remove the hot mirror and substitute a filter that **blocks visible** light, and the same sensor images by reflected **near-IR** (`fig-infrared-photography`, author's near-IR panorama on a full-spectrum-converted DSLR).

• **why it looks alien**: chlorophyll reflects near-IR strongly → foliage glows white (the **Wood effect**); the clear sky is blue only because air scatters short wavelengths (**Rayleigh ∝ 1/λ⁴**), so it scatters near-IR far less and collapses to a **deep near-black** — infrared's signature dark sky, strongest away from the sun, throwing white foliage/clouds into relief; skin smooths. Same reflectance underlies **NDVI** remote sensing, art-conservation/forensic imaging, and IR-illuminated security/scientific cameras.

• **the lesson**: the detector's spectral response is set by **physics** (silicon's band gap) and only *trimmed* to the visible by a filter we can choose to remove.

• **CCD vs CMOS**: a **CCD** shuttles charge across the chip in a "bucket brigade" to **one** amplifier at the edge (clean, uniform, but slow and power-hungry, and it reads the whole frame at once → naturally **global** shutter); **CMOS** puts an amplifier (and now the ADC + logic) **at every pixel** and reads in place, **row by row** (fast, cheap, low-power — now utterly dominant). [CCD bucket-brigade vs CMOS read-in-place schematic]

• what CMOS's per-pixel, row-by-row readout **unlocks** (and costs): on-chip ADC, region-of-interest and high-frame-rate readout, on-sensor HDR and PDAF and binning — but, because rows are read at different times, an **electronic rolling shutter** with its motion artifacts (below).

• the pixel value is an **integral over four dimensions** (aperture × pixel area × wavelength × **time**) — the [[#Image measurements as integrals]] chapter owns the first three; **this is the time one**. Over the **exposure time** Δt the well **accumulates** photon arrivals: value ∝ ∫ irradiance dt. [photons-accumulating-in-a-well figure]

• **exposure = irradiance × time** (the other half of the exposure triangle with aperture; ISO is gain *after*). **Reciprocity**: ½ the light for 2× the time gives the same exposure — until it doesn't (sensor reciprocity holds far better than film's reciprocity *failure*).

• the time integral is exactly **motion blur**: anything that moves during Δt is averaged along its path → a long exposure smears motion (light trails, silky water), a short one freezes it. The deliberate use and the *removal* of this integral are [[Motion blur, temporal sampling, and resampling]] and deblurring.

• **what ends the integral is the shutter** — which is the rest of this chapter.

• **mechanical shutter** — a physical curtain (or leaf) opens then closes to time the exposure. A focal-plane shutter is **two curtains**: at short times the second starts closing before the first fully opens, so a **slit** sweeps the frame (already a kind of rolling exposure, in hardware). A **leaf shutter** (in the lens) opens from the centre → exposes the whole frame at once (global) and **syncs flash at any speed**.

• **electronic shutter** — no moving parts: the sensor **resets** each photosite to start its integration and **reads it out** to end it. Silent, vibration-free, and capable of very short or very high-frame-rate exposures — but on CMOS it is usually **rolling** (below). Many cameras use an **electronic first curtain** (electronic start, mechanical end) to cut shutter shock while keeping a clean finish.

• **global shutter**: every pixel integrates over the **same** time window (all start and stop together), then the frame is read out. No motion distortion. Native to CCD and to special global-shutter CMOS; costs transistors/area per pixel.

• **rolling shutter**: rows are exposed and **read out one after another**, so each row's time window is **shifted** slightly later than the one above. Cheap and standard on CMOS — but rows see the scene at different instants. [row-vs-time diagram: global = a rectangle in (row, time); rolling = a slanted parallelogram]

• the **artifacts** (all from "different rows, different times"): **skew** (a fast pan or a moving car leans over), **wobble / "jello"** (handheld video shimmer), the **spinning-propeller / guitar-string** surreal shapes, and **partial exposure with flash** (a brief flash lights only the rows open at that instant → a **bright band**, which is why flash sync speed exists). [📺 [rolling shutter explained](https://www.youtube.com/watch?v=dNVtMmLlnoE) — Smarter Every Day; `fig-rolling-shutter` interactive: a vertical rod / spinning blade under global vs rolling readout]

• **lighting & flicker banding**: mains-powered and many **LED/PWM** lights **flicker** (100/120 Hz, or fast PWM dimming); under a rolling shutter, rows captured at different phases of that flicker get different brightness → horizontal **banding** (and colour banding if RGB phosphors lag). Fixes: match the exposure to the flicker period, **anti-flicker** modes that time the readout, or a global shutter. Ties to video [[Video stabilization and rolling-shutter correction]].

• **correcting rolling shutter** computationally — estimate the per-row time offset and the camera/scene motion, then warp each row back to a common instant → [[Video stabilization and rolling-shutter correction]].

• **off-axis (cos⁴) falloff** still applies once there's a sensor + finite aperture: rays reaching the frame edge arrive obliquely → **cos⁴θ** dimming toward the corners (**natural vignetting**), distinct from optical vignetting (Optics).

• the conventional sensor **couples** all four axes into one number; the **ideal sensor would decouple** them — measure each separately, combine *after* the shot

• a branch of comp-photo **slices/dices the integral** for a-posteriori image formation: **burst imaging** splits **time** (align, then sum → motion-blur control; → [[Multiple exposure imaging]]); **plenoptic/light-field** splits the **aperture** (DoF + focus after the shot; → [[Light fields and plenoptic cameras]]); **hyperspectral** splits **wavelength** (full spectrum/pixel); **polarization** filters at **multiple angles** split a fifth axis (specularities/reflections, glare, reflection separation). Move = *don't integrate yet; record the pieces, decide later*.

• **dual conversion gain (DCG)** — a CMOS pixel can switch the **sense-node capacitance**, changing the conversion gain (µV per electron): **high-conversion-gain** mode (small capacitance → lower read noise, better low light / high ISO) vs **low-conversion-gain** mode (large capacitance → bigger full well, more highlight headroom). Two settings → two **native ISOs** (**dual native ISO**); reading/combining both extends dynamic range → [[Noise, signal-to-noise ratio and dynamic range]].

• **quad-Bayer (Tetracell / quad CFA)** — the CFA groups **2×2 same-color photosites** under one filter tile. Good light: the sensor **remosaics** back to full-resolution Bayer; low light: it **bins** each group into one larger effective pixel (less noise, ¼ resolution). How 48/50/108 MP phone sensors get clean low-light output and in-pixel staggered-exposure HDR. Variant: **3×3 Nonacell**. → [[Demosaicking]], [[HDR merging]].

• **dual-pixel and quad-pixel autofocus (on-chip lens, OCL)** — split a photosite into **2 (dual-pixel)** or **4 (quad-pixel, 2×2 OCL)** sub-photodiodes under **one microlens**; each sub-diode sees a different half/quadrant of the lens pupil → **phase-detection autofocus (PDAF) at every pixel** (a stereo baseline inside the pixel); sub-diodes are **summed** for the final image (no light loss). Sensor structure only — the optics/AF algorithm are in [[Autofocus]].

• recap [[#Image measurements as integrals]]: a pixel value is an integral of the plenoptic function against a sampling kernel. Now stack everything: the **scene** becomes a vector $\mathbf{x}$, the **image** a vector $\mathbf{y}$, and the imaging system is a **matrix** $A$ — $\mathbf{y}=A\mathbf{x}+\mathbf{n}$. Each **row** of $A$ = one pixel's kernel (which scene elements it sums, with what weights).

• the big-lesson tie-in: **image formation is selecting and combining rays** — the pinhole *selects* a single ray per pixel, the lens *combines* a cone; $A$ is exactly that selection/combination, written down. (→ [[Big Lessons]]; [[Pinhole image formation and linear perspective]]; [[Lens image formation]].)

• **flatland**: a 2-D world (a 1-D scene, a 1-D sensor), both **discretized**, so $A$ is a small drawable matrix. Two standard ways to feed $A$: a **billboard** (next section) or **ray space** (the section after).

• input = a **flat 2-D billboard** at a chosen depth — the scene as a 1-D row of coloured points; $A$ maps billboard → sensor. Show each system's flatland $A$ as a heat-map [`fig-linsys-flatland`, `fig-linsys-matrices`]:

• **pinhole** — each sensor pixel sees **one** billboard point → $A\approx$ a permutation/identity (sharp; throws away nearly all light).

• **pinspeck** (anti-pinhole / accidental occluder) — the **complement** of a pinhole: $A=$ all-ones minus the pinhole row pattern (Cohen's accidental cameras).

• **lens** — each pixel integrates a **range** of billboard points (the defocus disc off the focal plane): $A=$ a **banded / blur** matrix; $\approx$ identity at the focal depth, wider off it.

• **corner camera** — an occluding edge turns the scene into a **cumulative integral**: $A\approx$ lower-triangular; differentiating the measurement recovers angle (penumbra / accidental cameras).

• **stereo pair** — two pinholes → two images stacked: $A$ is **taller** (more rows); the two viewpoints give disparity → depth.

• **coded aperture** — a patterned mask in the aperture: $A=$ a **coded blur** engineered to be broadband / zero-free (invertible) and depth-revealing (**forward-ref** [[16 Computational optics and coded imaging|Coded aperture]]).

• **the limitation**: the billboard $A$ **depends on depth** — move the billboard and $A$ changes (the lens blur widens, the stereo disparity shifts). One matrix describes only one depth. That is exactly what the ray-space view fixes.

• input = the **light field**: the set of **all rays** arriving at the system (ray space — [[14 Light fields and plenoptic cameras|Part 13]]). The imaging system is then a **fixed** linear projection of ray space → image, **independent of scene depth** (depth becomes structure *within* $\mathbf{x}$, not a change to $A$).

• the **same cameras** re-expressed as projections of ray space: a pinhole = a thin slice; a lens = an integral along a sheared line; a coded aperture = a coded projection; a stereo / light-field camera = **more rows** (more of ray space measured). [`fig-camera-tradeoffs-1`, after [@levin-etal-2008-lightfield]]

• cite [@torralba-etal-fcv] (the camera as a linear system) and [@levin-etal-2008-lightfield] (trade-offs via light-field projections).

• the idea to land: **much of computational photography and computational optics is the search for new linear imaging systems $A$ — with matching reconstruction methods — that extract better images and extra information, notably depth.**

• extend $\mathbf{x}$ with **time**: a moving scene under an exposure is again a linear system, now over **space–time**. The exposure integrates along the motion trajectory (motion blur = a linear operator whose kernel is the path).

• **motion-invariant photography** ([@levin-etal-2008-motion]): sweep the camera along a **parabolic** trajectory during the exposure so the blur PSF is the **same for every object speed** → a single deconvolution deblurs all of them. Designing the *operator* (the parabolic sweep) turns an otherwise object-dependent $A$ into one invertible kernel. [`fig-linsys-motion-invariant`]

• the temporal sibling of coded aperture / wavefront coding — *engineer the operator so it inverts cleanly*; full treatment in [[16 Computational optics and coded imaging]] and [[Motion blur, temporal sampling, and resampling]].

• the clean $\mathbf{y}=A\mathbf{x}$ is an idealization; a few real departures matter.

• **clipping / saturation** — wells and code values have ceilings; past saturation the response flatlines. A hard nonlinearity that **destroys information** (no inverse) — poisons deconvolution and exposure merges, so saturated pixels must be detected and handled (→ [[Blind deblurring]], [[HDR merging]]).

• **gamma & quantization** — encoding nonlinearities, both benign. **Gamma** is an *invertible* power law $V=L^{1/\gamma}$: decode to work in linear light, or operate directly on encoded values — exposure becomes $V\mapsto 2^{EV/\gamma}\,V$ and a white-balance gain $g_c$ becomes $V_c\mapsto g_c^{1/\gamma}\,V_c$ (exact for a pure power law; ≈ for sRGB's linear toe). **Quantization** is best **modeled as additive noise** (uniform, variance $\Delta^2/12$ for a code step $\Delta$), dwarfed by photon/read noise at any sane bit depth — it matters only where it compounds (repeated 8-bit round-trips, hard tone curves → banding, the cue for dithering). (→ [[Measuring and encoding color|Color technology]].)

• **not everything we want is linear in the scene** — depth, motion, material, identity are nonlinear read-outs of the light, not matrix inverses (→ [[Limitations of the medium]]).

• **a linear forward model ≠ a linear solution** — recovery leans on priors/inductive biases, so the map from measurement to estimate is thoroughly nonlinear. *Linear physics, nonlinear inference.*

• the question that organizes everything after this chapter: **how invertible is $A$?** — how well can you recover $\mathbf{x}$ (a sharp image, a depth map, more) from $\mathbf{y}=A\mathbf{x}+\mathbf{n}$?

• two complementary ways to get **more invertible information**:

• **coded imaging / computational optics** ([[16 Computational optics and coded imaging]]): keep the measurement count but **redesign $A$ to be better-conditioned** — coded aperture, wavefront coding, the **lattice-focal lens** ([@levin-etal-2009-4dfreq]) — so the same data inverts more cleanly (and reveals depth).

• **light fields & multiple-exposure photography** ([[14 Light fields and plenoptic cameras]], [[11 Multiple exposure imaging]]): **make $A$ taller** — take *more* measurements (more views, more exposures) — so the system is more determined and carries more information.

• the tool for *reasoning* about invertibility is a **change of basis that diagonalizes $A$** — most often **Fourier analysis** (convolution → per-frequency multiplication), which makes the conditioning (where $A$ has small / vanishing singular values) and the inversion legible. (→ [[Linearity, Fourier, Aliasing and deblurring]]; [@levin-etal-2009-4dfreq] do exactly this 4-D-frequency analysis for computational cameras.)

• two chapters, one line. [[#Imaging as a linear system]] said a camera is a **linear projection**, $\mathbf{y}=A\mathbf{x}$; [[#Noise, signal-to-noise ratio and dynamic range]] said the sensor **adds noise**. Put them together and you get the **fundamental equation of imaging**: $$\mathbf{y}=A\mathbf{x}+\mathbf{n}.$$ $\mathbf{x}$ = the scene/image we want, $A$ = the imaging operator (blur, projection, sub-sampling, masking…), $\mathbf{n}$ = noise. [`fig-inverse-forward-model`]

• the **forward** direction (scene → measurement) is what the camera does and what we now understand. This chapter is about the **backward** direction — **given $\mathbf{y}$ (and a model of $A$), recover $\mathbf{x}$** — the **inverse problem**. Deblurring, super-resolution, demosaicking, inpainting, denoising, tomography, refocusing, depth-from-defocus: *all the same problem*, only $A$ changes.

• **why it's hard: information is lost — twice over.** (1) **the $+\mathbf{n}$**: noise corrupts the measurement, so even a perfectly invertible $A$ gives a noisy answer. (2) **the $A$**: a real $A$ is **not nicely invertible** — it can be **rectangular** (fewer measurements than unknowns → it literally throws information away) or **square but ill-conditioned** (it *crushes* some components of $\mathbf{x}$ to nearly zero, so undoing it *amplifies* whatever noise sat there). Either way, **naively "dividing by $A$" is a disaster.**

• **ill-posed** (Hadamard): a problem is *well-posed* if a solution **exists**, is **unique**, and depends **continuously** on the data. Imaging inverse problems routinely fail all three — no exact solution (noise), not unique (null space), and not continuous (tiny measurement noise → huge swings in the naive estimate). **The only cure is extra knowledge** about what $\mathbf{x}$ should look like. The rest of the chapter is two ways to supply it. [@hadamard-1902]

• **two perspectives, one chapter**: the **linear-algebra** view (§1 — condition the problem, then **regularize**) and the **probabilistic/Bayesian** view (§2 — a **likelihood** for the measurement times a **prior** for the scene). They turn out to be the **same thing** when the noise is Gaussian and you take logarithms (§2, §4) — but the Bayesian view is more general and says *where the knowledge comes from*. We end (§3) on what the prior actually is — the **statistics of natural images** — and (§4) recap, including the **big shift** generative AI has caused.

• the linear-algebra question is blunt and useful: **look at $A$ and ask how badly it's going to behave** before you try to invert it. Two things to read off — its **shape** and its **conditioning**. [`fig-inverse-shapes`, `fig-inverse-conditioning`]

• **shape / rank — is it even determined?**

• **square & full-rank**: as many measurements as unknowns, in principle invertible — but see *conditioning* below.

• **tall (overdetermined)**: **more** measurements than unknowns (e.g. burst / multi-shot). No exact solution through all the noisy data, but **least squares** $\hat{\mathbf{x}}=\arg\min\|A\mathbf{x}-\mathbf{y}\|^2$ **averages the noise down** — *more measurements is the cleanest cure.* (→ [[11 Multiple exposure imaging]].)

• **wide (underdetermined)**: **fewer** measurements than unknowns (super-resolution, inpainting, **demosaicking**, compressive sensing). $A$ has a **null space** — a whole subspace of $\mathbf{x}$ that maps to the *same* $\mathbf{y}$ — so the data alone **cannot** pick the answer. Something else must. (This is exactly where a **prior** becomes non-negotiable.) *Everyday example*: a **Bayer mosaic** records one colour per pixel → **2/3 of the rows we need are missing**; demosaicking fills them only via a prior.

• **conditioning — even "invertible" can be useless.** A square $A$ can be technically invertible yet **ill-conditioned**: its **singular values** range from big to nearly zero (condition number $\kappa(A)=\sigma_\max/\sigma_\min$ huge). Inverting multiplies each component of the measurement by $1/\sigma$; the near-zero $\sigma$'s blow up the noise sitting on those components. A **lens blur** is the textbook case — it kills high frequencies (tiny singular values there), so naive deblurring detonates into noise. [`fig-inverse-conditioning`]

• **regularization — tilt the answer toward plausibility.** Instead of $\arg\min\|A\mathbf{x}-\mathbf{y}\|^2$ (which chases the noise), solve $$\hat{\mathbf{x}}=\arg\min_{\mathbf{x}}\ \underbrace{\|A\mathbf{x}-\mathbf{y}\|^2}_{\text{data fit}}+\lambda\,\underbrace{R(\mathbf{x})}_{\text{regularizer}}.$$ $R$ **penalizes implausible $\mathbf{x}$** (wiggly, not smooth, not sparse…) and **$\lambda$** trades data-fit against plausibility. **Tikhonov** ($R=\|\mathbf{x}\|^2$ or $\|\nabla\mathbf{x}\|^2$) damps the blow-up on the small singular values; **total variation** ($\|\nabla\mathbf{x}\|_1$) keeps edges. This single line — *data term + regularizer* — recurs through deblurring, SR, inpainting, tomography. [@tikhonov-1963; @bertero-boccacci-1998]

• **better numerical tools vs a fundamentally hard problem — be honest.** Some difficulty is **numerical** (a big sparse system, slow or unstable to solve) and yields to **better solvers** — conjugate gradient, multigrid, FFT-diagonalization, preconditioning ([[05 Computational tools#Efficient solvers]]). But **conditioning and rank are properties of $A$ itself**: no solver recovers what the null space erased or the noise drowned. *Distinguish "hard to compute" from "impossible to recover."* The first is engineering; the second needs **more data or a better prior** — full stop.

• **sometimes you don't even know $A$** — and sometimes you *think* you do but you're **wrong**.

• **unknown $A$ (blind problems)**: blind deblurring estimates the **blur kernel and the image together**; you're solving for $A$ *and* $\mathbf{x}$ at once — much harder, and again rescued only by priors (on both). (→ [[09 Single-image computational photography#Blind deblurring]].)

• **wrong $A$ (model error)**: the more insidious case — your $A$ is *almost* right (the PSF is slightly off, the noise isn't quite Gaussian, the optics drift). A confident inversion of a subtly-wrong model produces confident **artifacts**. Robustness to model error is a real and under-appreciated cost.

• **forward-ref — the magic basis**: all of this conditioning talk gets **dramatically clearer when $A$ is diagonal**, because then each component inverts independently and you can *see* exactly which ones are doomed. For the convolutions that dominate imaging, the **Fourier basis diagonalizes $A$** — deblurring becomes per-frequency division, and the ill-conditioning is simply *the frequencies the blur zeroed out*. That is the subject of [[Linearity, Fourier, Aliasing and deblurring]] (and the **Wiener filter**, the regularized per-frequency inverse, [@wiener-1949]).

• so far $A$ is **given**; but in computational photography $A$ is **often ours to choose** (coded aperture, microlens array, coded exposure, end-to-end-optimized lens). Then the problem flips: not "invert this $A$" but **"design the $A$ — with its reconstruction — for best end-to-end performance."** Acquisition + recovery = one jointly optimized system; pick the operator whose measurements carry the most **recoverable information** (coded imaging makes $A$ better-conditioned; light fields make $A$ taller → [[16 Computational optics and coded imaging]], [[14 Light fields and plenoptic cameras]]).

• **teaser → [[05 Computational tools#Information theory]]**: the tool for "which $A$ is best?" is **information theory** — read the imaging system as a **noisy band-limited channel** with **capacity** $C=\int\log_2(1+\mathrm{SNR}(f))\,df$ = how many bits of scene a given $A$ can deliver; maximize it, then reconstruct. **Flag its reach**: information theory is strong on the **measurement-and-transport** side (capture/store/move — channel, compression, detector efficiency) and **silent on the perceptual/aesthetic** side (nothing about what makes a photo *look* good). Governs recoverable information; stops where taste begins.

• same problem, **probabilistic** retelling — and the more fundamental one. We don't want *an* $\mathbf{x}$ that fits; we want the **most probable $\mathbf{x}$ given the measurement**, combining **what we observed** with **what we expected**. Bayes' rule: $$\underbrace{p(\mathbf{x}\mid\mathbf{y})}_{\text{posterior}}\ \propto\ \underbrace{p(\mathbf{y}\mid\mathbf{x})}_{\text{likelihood}}\ \cdot\ \underbrace{p(\mathbf{x})}_{\text{prior}}.$$ [`fig-inverse-bayesian`]

• **three ingredients, kept apart**: every imaging inverse problem = a **forward model** ($A$), a **noise model** ($\mathbf{n}$), and a **prior** ($p(\mathbf{x})$). The first two *are* the likelihood; the third is the prior. Naming them is what lets you swap one without touching the others (new camera → only $A$; noisier sensor → only the noise model; better image model → only the prior).

• **the likelihood** $p(\mathbf{y}\mid\mathbf{x})$ is **the forward model plus the noise model**: it scores how well a candidate $\mathbf{x}$ explains the data. For $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ with Gaussian noise, $p(\mathbf{y}\mid\mathbf{x})\propto\exp(-\|A\mathbf{x}-\mathbf{y}\|^2/2\sigma^2)$ — the **data-fit term** in disguise.

• **two cautions about "linear"** (the whole picture leans on it): (1) *linear in **what**?* $\mathbf{y}=A\mathbf{x}$ is linear in **scene radiance** (light adds), **not** in **encoded pixel values** — images are usually **gamma-encoded** (nonlinear), which breaks the model; **work in linear light**, undo gamma first (→ [[04 Basic image processing and ISP#What the numbers actually mean: encoding|gamma, BASICS]]). (2) a **linear forward model ≠ a linear solution**: the scene→measurement map is linear, but the measurement→estimate map $\mathbf{y}\mapsto\hat{\mathbf{x}}$ is generally **nonlinear** (the regularizer/prior, clamps, sparsity make it so). *Linear physics, nonlinear inference.*

• **the prior** $p(\mathbf{x})$ is **everything we know before looking** — that natural images are smooth-ish, edge-y, not random noise. It is the principled relative of the regularizer: $R(\mathbf{x})\leftrightarrow-\log p(\mathbf{x})$. Where regularization felt like a numerical *trick* ("add a penalty to behave"), the prior is a **statement about the world** you can argue about, measure, and improve — *philosophically more grounded, and more general* (it can be non-Gaussian, multimodal, learned).

• **the question**: we now have a forward model (linear or not), a noise model, and a prior — together a **posterior** over plausible scenes. But a posterior is a *distribution* and a user wants **one image**, so: *given the distribution of plausible solutions, what single answer do we return?* This is where **inference, optimization, and loss** all enter — a small menu, and a real choice.

• **pick the most probable — MAP.** the **peak** of the posterior: take **$-\log$** and **minimize**, $$\hat{\mathbf{x}}_{\text{MAP}}=\arg\min_{\mathbf{x}}\ \tfrac{1}{2\sigma^2}\|A\mathbf{x}-\mathbf{y}\|^2-\log p(\mathbf{x}).$$ This is where **inference becomes optimization** — and with a **Gaussian prior**, $-\log p(\mathbf{x})$ is a **quadratic** and the whole thing *is* **regularized least squares (Tikhonov)** ($\lambda\propto\sigma^2$): **regularization = MAP under a Gaussian prior**, same equation two stories. The $-\log$ turns a *product* of probabilities into a *sum* of penalties — data term + prior/regularization term. [`fig-inverse-bayesian` inset]

• **minimize an aggregate distance — which distance?** instead of the peak, return the answer **closest on average** to the truth. Squared error → the **MMSE / posterior mean** (the average; what an optimal **denoiser** returns → [[Image priors]], **Tweedie**). But "closest" begs **which distance / loss?** squared-pixel $L_2$ is *convenient* (linear derivative), not *right* — perceptually naive; a **perceptual** loss (SSIM, LPIPS, deep features) is closer to the eye but harder to optimize / easier to game (→ [[Image metrics]]; perceptual & adversarial losses in [[Deep learning]]). *The loss states what "right" means.*

• **sample them — posterior sampling.** abandon a single best estimate: **draw one** plausible scene from the posterior (diffusion posterior sampling & kin) — sharp, self-consistent, **on** the manifold (where an average falls **off** it); often the most *natural-looking* answer even when not the most probable / closest-on-average.

• **not interchangeable**: MAP/MMSE **coincide for a Gaussian** posterior, **diverge** for skewed/multimodal (heavy-tailed) ones — MAP on a sharp peak, the mean nowhere near it, a sample elsewhere on the manifold. The right one depends on what error you pay for ([@kay-1993]).

• **better with a prior, always (L5.4)**: regularizer or prior, loss or likelihood — for an ill-posed problem the prior is **not optional**, the only thing between you and noise.

• **a lucky Bayesian case**: in photography **data is cheap** — priors are easy to build/learn from billions of images (unlike **black-hole imaging** / Event Horizon Telescope: one precious, radically incomplete measurement, ~no ground truth). Caveat: **fully** ground-truthed data is rarer than it looks — even **demosaicking** has no easy "true" full-colour image behind the Bayer mosaic.

• **priors are often implicit**: a bilateral filter, edge-aware/gradient demosaicking, an end-to-end network — none *names* a prior, yet each **embeds one** (a bet about natural images). The bet is made regardless; the only question is whether on purpose. **The Bayesian virtue**: it cleanly **disentangles forward model / noise / prior / inference** — modular, theoretically cleaner (improve the prior → every task improves; → [[Image priors]]). But the clean factoring **isn't always possible and can leave performance on the table** (a tightly-coupled end-to-end method may just work better). The next section asks what that prior actually *is*.

• a prior is only as good as its grip on **what real images look like** — so what *do* they look like, statistically? The key picture: in the space of all possible pixel arrays, **almost everything is noise**; **natural images** are an astonishingly **tiny, structured subset**. [`fig-natural-image-manifold`]

• **the manifold of natural images.** A megapixel image is a point in a million-dimensional space. **Random** points in that space are TV-static **noise**. The images a camera ever produces lie on a **thin, curved, low-dimensional manifold** winding through it. A good prior $p(\mathbf{x})$ is **high on that manifold and ~zero off it** — it *is* a description of the manifold. Recovery = "of all $\mathbf{x}$ consistent with $\mathbf{y}$, snap to the manifold."

• **what we know about the manifold** (measurable regularities a prior can exploit): natural images have **$1/f$ amplitude spectra** (scale-invariance — [@field-1987; @ruderman-bialek-1994]); **heavy-tailed gradient / wavelet statistics** (mostly flat, rare sharp edges → **sparsity**, the basis for [@olshausen-field-1996] sparse coding); strong **local correlation**. A simple *generative* model produces all of this from **occlusion alone** — the **dead-leaves model** (opaque shapes of random size piled up, power-law sizes): it yields the $1/f$ spectrum + scale-invariance + heavy-tailed edges ([@lee-mumford-huang-2001]; Ruderman 1997 [@ruderman-1997]; Matheron), the hand-built ancestor of learned generative priors (diffusion). These regularities are *exactly* what TV, wavelet-sparsity, and learned priors encode. ([@simoncelli-olshausen-2001; @hyvarinen-etal-2009] survey the whole story.)

• **from hand-written to learned.** Early priors were **hand-designed** and **coarse** (smoothness, TV, a few wavelet coefficients) — easy to write, weak at capturing the manifold's real shape. Modern priors are **learned** from millions of images and capture it astonishingly well. **Forward-ref + teaser**: the strongest learned prior we have is a **generative model** — a diffusion model is, precisely, a learned model of $p(\mathbf{x})$ (the natural-image manifold), and it is the subject of the **last chapter of [[05 Computational tools]]**, [[05 Computational tools#Generative AI and diffusion]]. (The cash-out — *any denoiser is a prior*, PnP/RED, diffusion ≈ iterated denoising — is [[Image priors]].)

• **inductive bias vs prior vs data** (💡): knowledge enters as an explicit **prior** (a penalty you write), an **inductive bias** (assumptions in the *machinery* — wavelet basis, a CNN's locality/translation-equivariance, the architecture), or **data** (fit to millions of examples). The arc shifts weight from the first two to the third — a local case of **Sutton's "bitter lesson"** [@sutton-2019]: general methods that **scale with compute + data** keep overtaking hand-engineered knowledge (a learned model that *saw* the manifold beats any penalty *describing* it). Caveat: bitter lesson = what wins **at scale**, not what's **interpretable / data-efficient** — hand-built priors + inductive biases still win when data is scarce, guarantees matter, or you need to know *why* (scientific/medical imaging).

• **the whole chapter on one slide.** $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ is **hard for two reasons**: **(1) noise** ($+\mathbf{n}$) corrupts the data; **(2) the operator $A$ loses information** — it's **rectangular** (a null space the data can't fill), or **non-invertible / ill-conditioned** (singular values at or near zero, so inversion amplifies noise). Recovery therefore **needs extra knowledge** — a regularizer or, equivalently, a prior.

• **denoising is the quintessential inverse problem** — and it sits at **one end of the space of inverse problems**: take the *purest* case, $A=I$ ($\mathbf{y}=\mathbf{x}+\mathbf{n}$) — the operator is as benign as possible, **nothing to invert**, yet recovery is *still* impossible without help. Cleanest proof the difficulty was never only about undoing a hard operator: the **noise alone** makes it ill-posed (the *other* end is where $A$ is the villain — rectangular / ill-conditioned). The **measurement term does nothing** — every $\mathbf{x}$ fits the data equally as far as $A$ is concerned — so **the prior does 100% of the work**. Without extra knowledge about natural images you **cannot denoise at all**; with it you can. Denoising is the inverse problem stripped to its core, which is why "*any good denoiser is a universal prior*" became the organizing idea of modern restoration ([[Image priors]]). And it exposes the **average-is-dull** trap: the **MMSE** estimate, averaging plausible on-manifold images across a curved manifold, lands **off** it and **blurry** — optimal in squared error, not a valid image. [`fig-natural-image-manifold`, `fig-denoising-without-prior`]

• **the two perspectives, reconciled (one more time).** Linear-algebra **regularization** and Bayesian **prior** are the **same** when the noise is Gaussian, the observation is **linear**, and you take **$-\log$** and maximize: regularized least squares **=** MAP with a Gaussian prior. The Bayesian frame stays the more general one — non-Gaussian priors, the MAP-vs-MMSE choice, learned $p(\mathbf{x})$ — and it tells you *where* the regularizer came from.

• 💡 **the big trend (where this is going).** Historically the two halves of $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ got **uneven attention**: we became **very good at the measurement side** (precise forward models $A$, exact least-squares / Fourier / Wiener inversion) while priors stayed **coarse** (smoothness, TV). **Generative AI flipped the balance**: diffusion models give us **astonishing priors** — a near-perfect grip on the natural-image manifold — but it has proven **hard to keep their output faithful to the measurement** ($\|A\hat{\mathbf{x}}-\mathbf{y}\|$ can drift; the model "hallucinates" plausible-but-wrong detail). The live research frontier is **honouring the data *and* the learned prior at once** (diffusion posterior sampling, DDRM, PnP with learned denoisers — [@kadkhodaie-simoncelli-2021; @chung-etal-2023-dps; @kawar-etal-2022-ddrm]).

• **hand-off**: every later restoration chapter is a special case of this one — *pick $A$, pick a prior, balance data-fit against it.* The **solvers** are [[05 Computational tools#Linear Inverse Problems and Regression]]; the **Fourier** view that makes conditioning legible is [[Linearity, Fourier, Aliasing and deblurring]]; the **priors** (hand-built → learned → diffusion) are [[Image priors]] and [[05 Computational tools#Generative AI and diffusion]].

• the medium is **specific**: a picture is **flat**, **finite** (it has a frame), shows **one viewpoint**, is usually **static**, and has **limited contrast** and **limited gamut**. The world the eye sees is none of these. Each mismatch is a **limitation of the medium**, and the picture maker has exactly three responses: **compensate** (push a surviving cue harder so it stands in for the missing one), **accentuate** (lean into what the medium does well), or deliberately set cues in **conflict** for effect (Magritte's scale games, Escher's impossible figures). [@durand-2002]

• the deep framing (NPAR 2002): depiction is *"the inverse of an inverse problem."* Vision already *infers* the world from the optical flow on the retina; a picture must produce a **new** optical flow — within the medium's limits — that drives the *same* inference. So *"the direct recording of the optical flow (i.e. photography) might not result in the most realistic image"*: sometimes you must **reinforce the occluding contour** to make up for absent stereo, or **add flare** to suggest a brightness no pigment can emit. [`fig-medium-six-limitations`]

• this chapter is the **map of the limitations**; the rest of the book is the **toolbox** that attacks them — HDR for contrast, light fields and stereo for flatness, video for time, panorama for the frame, NPR for accentuation on purpose.

• the first and richest limitation: a picture **collapses the third dimension**. We perceive depth from a long list of **cues**, and the crucial split is *which ones a flat picture can carry and which it cannot* (Helmholtz's **unconscious inference** — the brain *infers* depth from these cues). [`fig-depth-cue-catalogue`]

• **cues a flat picture CANNOT reproduce** (they need two eyes, a focusing eye, or a moving viewer):

• **binocular disparity / stereopsis** — the two eyes' slightly different images, decoded into depth (Julesz's **random-dot stereogram** proves it works on disparity alone, with no monocular form). A flat print shows both eyes the same image. [@julesz-1971]

• **convergence** — the inward rotation of the eyes to fixate; useful only out to a few metres.

• **accommodation** — the muscular focusing of the eye's lens; a weak cue, close range only. (Its *visual* shadow, **defocus blur**, *does* survive — see below — but the proprioceptive cue is gone.)

• **motion parallax** — near things sweep past faster than far things when the viewer moves; a still photo can't deliver it (a video can → [[13 VIDEO]]).

• **why a flat picture still reads as 3-D — the pseudoscope.** A pseudoscope swaps the eyes' images, *reversing* stereo; the brain **ignores the reversed disparity and trusts the pictorial cues** (occlusion, shading). Lesson: **pictorial cues are stronger than stereo** — which is exactly why a painter or photographer can fake depth convincingly on a flat surface.

• **pictorial cues a flat picture CAN reproduce** (the painter's toolkit — and what good depiction over-emphasizes to compensate for the missing cues above):

• **occlusion / overlap** — the strongest, most robust cue; purely ordinal (front/behind). (Lascaux's overlapping animals; figure–ground.)

• **relative size** — equal objects shrink with distance.

• **familiar size** — known real sizes set distance (and let you *lie* — Magritte's room-filling apple, the forced-perspective giant).

• **height in the visual field** — bases higher in the frame (toward the horizon) read as farther (Bruegel; Chinese scrolls; medieval stacking).

• **linear perspective** — parallel lines converge to vanishing points (Raphael's *School of Athens*); the geometry of [[#Pinhole image formation and linear perspective]].

• **shading** + the **light-from-above prior** — gradients model form; the brain *assumes light comes from above*, so shaded bumps flip to dimples (and craters to domes) when inverted — the **hollow-face** reversal. A flat picture exploits this hard-wired assumption. [`fig-light-from-above`]

• **cast shadow** — anchors objects to the ground and signals height (Kersten–Mamassian **ball-in-a-box**: identical balls rest or float depending only on the shadow). [@kersten-etal-1997]

• **texture gradient** — a uniform texture compresses with distance, giving slant and recession (Constable; a receding gravel field).

• **aerial / atmospheric perspective** — distant things lose contrast and saturation and turn bluish (Leonardo's hazy backgrounds); also a **grouping** cue (things at one distance share a tint).

• **defocus blur (depth from defocus)** — a photograph can **bake in** depth of field (sharp subject, soft background) as a pictorial cue — but note it is the photographer's *choice* and **denies the viewer the ability to refocus** (the through-line to [[Photographs are usually not passive objective recordings]] and to [[14 Light fields and plenoptic cameras]], which gives refocus *back*).

• **conflict / dissonance.** Because pictorial cues are independently controllable, you can set them against each other on purpose — the hollow-face reversal, Escher's locally-consistent-but-globally-impossible figures, Magritte's scale. The medium's flatness is not only a loss; it's a **license**.

• a single photograph **freezes one instant** — duration, change, and motion are lost. Yet a still image conveys movement through a whole vocabulary, split into *within one frame* vs. *combining several*.

• **within a single snapshot:**

• **telling content & pose — deviation from rest.** A body caught *away from its resting position* reads as in motion (it "wants" to fall back): the skater mid-leap, Géricault's "flying gallop," the *Discobolus* vs. a static Egyptian figure. (Developmental note — the reading of action-pose is learned and *"the more dramatic, the better."*)

• **motion blur** — the streak a moving subject leaves in a finite exposure; a painter anticipated it (Velázquez's blurred spinning-wheel spokes) long before the camera made it automatic.

• **trails** — long-exposure light-trails: car tail-lights, **Feininger's** helicopter rotor-tip arcs.

• **action lines / path** — the comic-strip convention of speed lines and drawn trajectories (Keith Haring), and dance-notation paths.

• **dynamic viewpoint** — a tilted, low, wide angle that adds energy.

• **combining several instants — chronophotography & capture.** **Muybridge's** sequential gallop plates (which settled whether a galloping horse ever has all four feet off the ground) and **Marey's** multi-phase chronophotographs and **Edgerton's** stroboscopic high-speed flash (the bullet through the apple, the milk-drop coronet) *capture* the time axis the single frame threw away. (These are the ancestors of [[11 Multiple exposure imaging]] and high-speed/video.) [@muybridge-1887]

• **depicting time on one surface — Futurism.** **Balla's** *Dynamism of a Dog on a Leash* and **Duchamp's** *Nude Descending a Staircase* paint multiple phases at once — *depiction* of motion rather than *capture* of it. [`fig-motion-three-ways`]

• the computational sequel is the whole **video** part: recover the lost dimension by sampling time densely (frame rate, motion blur, interpolation, video magnification → [[13 VIDEO]]).

• two coupled limitations. A picture is taken from **one fixed station point** (unlike the moving, binocular observer), and it has a **hard boundary** — a limited **field of view** — so everything outside is simply cut off.

• **the single viewpoint** *"cannot resolve ambiguity"*: from a **generic** viewpoint a shape reads correctly; from an **accidental** one it deceives (the forced-perspective gag). Picture makers smuggle *more* viewpoints in — mirrors (Velázquez's *Rokeby Venus*), engineering **multiple views** (plan/elevation), **exploded** and **divergent/reverse** perspective (Byzantine icons, Hockney's photo-joiners), and ultimately **Cubism** (many views of one face at once). The tension is **"what I see vs. what I know"** — Turner's *"I paint what I see"* against Picasso's *"I paint what I know."*

• **the frame.** A picture is a **window** (Alberti's *finestra aperta*; Dürer's perspective machine): a finite rectangle with a sharp edge the eye never experiences. The maker **chooses the cut** — composition, **cropping**, and the **panorama** (stitching a wide field to *escape* the frame — which needs **curvilinear** perspective, because linear perspective explodes past ~90°). [`fig-frame-and-fov`]

• **put numbers on it:** human vision spans ~**180–200°** horizontally (with only ~**2°** of sharp foveal vision, steered by saccades — [[#Human vision and color]]); a normal lens/print subtends maybe **40–50°**. The picture is a narrow, hard-edged **porthole** onto a panoramic, edgeless world. Field of view is also bound up with **focal length and perspective** ([[#Lens image formation]]): a wide lens crams in a huge field but distorts a near face; a long lens narrows the field and flattens depth.

• the **real world spans ~10⁻⁶ to 10⁶ cd/m²** — about **twelve orders of magnitude** of luminance across conditions — while a **print** renders at best ~1:500 contrast (often only ~1:50) and a display ~1–100 cd/m². The scene's enormous range simply **does not fit** the medium. [`fig-dynamic-range-gauge`]

• **two distinct problems:** (1) **intensity mismatch** — a sunny scene is *viewed in a dim room*; the picture's actual luminances live in a different, lower band than the scene's. (2) **insufficient contrast** — a high-dynamic-range scene must be **compressed** into the medium's narrow range.

• **why the eye beats the medium:** the visual system judges **local contrast / reflectance**, not absolute luminance — **local adaptation** and **center-surround** edge response (Land's **Retinex**), so we read both deep shadow and bright highlight that no single exposure holds. The medium has to *fake* that with tone curves and local edits. [@land-mccann-1971]

• **the historical acknowledgments of the limit:** Andrea Pozzo's painted ceiling *"is not bright enough"* to be a real sky; **Brunelleschi** left the sky in his perspective demo as **polished silver to reflect the actual sky**, because no pigment could reach its brightness.

• **the compensations (the modern toolbox in embryo):**

• **exposure & metering** — place the scene's range inside the medium (center-weighted, **spot**, **incident** metering; multi-segment "matrix" metering) ([[Exposure]]).

• **the Zone System** (Ansel Adams) — pre-visualize the scene's tones and *place* them in the printable range: the photographer's **manual dynamic-range compression** ($\approx$1:10,000 scene → 1:300 print). [@adams-negative]

• **dodge & burn** — locally lighten/darken in printing (W. Eugene Smith's "five days to print" Schweitzer); **edge burning** darkens borders to hold the eye — the manual ancestor of every **local tone operator**.

• **tone reproduction** — film/print's **toe–linear–shoulder** curve gently rolls off shadows and highlights; **gamma**. (→ [[04 Basic image processing and ISP]].)

• **perceptual & time-dependent tone mapping** — reproduce a *faithful impression* rather than physical values; simulate adaptation, glare, and slow dark adaptation (Ferwerda et al. 1996). The whole **HDR / tone-mapping** chapter is this compensation, automated. [@ferwerda-etal-1996]

• **flare, halo, glow** — adding bloom around a bright source *suggests* a brightness the medium can't emit — a learned "this is very bright" cue (NPAR's pictorial techniques). [@durand-2002]

• **gamut (the sibling limitation):** the medium also works from a **restricted set of primaries/pigments** that cannot reach all real colors — the color analog of limited contrast, handled by **gamut mapping** ([[#Color technology]]).

• one more axis, and the **optimistic** one. The eye's foveal acuity is about **1 arc-minute** (~60 cycles/degree); a picture is **uniformly sampled** at a fixed resolution. Of all the medium's limitations, **spatial resolution is the one capture and display have essentially caught up to** — high-megapixel sensors and "Retina"-class prints/screens reach or exceed foveal acuity at a normal viewing distance, so a still photograph can already be *indistinguishable from reality in sharpness*.

• this is the chapter's punchline: resolution is the **success story** that throws the *persistent* limitations — dynamic range, the third dimension, time, the finite frame, gamut — into relief. The rest of the book is, in large part, the ongoing campaign against *those*.

• the problem: **the same numbers look different on different devices** — each camera / monitor / printer has its own primaries, white point, gamut and transfer curve

• the solution — **ICC color management**: every device carries a **profile** mapping its values to/from a device-independent **profile-connection space** (XYZ or Lab); image → PCS → target device makes color *portable*

• **rendering intents** for out-of-gamut color: **perceptual** (compress the whole gamut, keep relations), **relative colorimetric** (clip + white-point match), **saturation**, **absolute colorimetric**

• **chromatic adaptation** between white points via a **Bradford 3×3** transform (the engineering cousin of von Kries adaptation, Perception)

• in practice: profiles embedded in files (→ File formats), monitor calibration, soft-proofing; **sRGB** as the safe default when no profile is present