array, basic representation and manipulation

• an image is an H×W×C array: a 2D grid of pixels, each a vector of C channels (usually R, G, B)

• sometimes extra channels, for example hyperspectral or for transparency — more in compositing chapter

• warn that it can be a problem when reading PNG if you assume just RGB and you get more channels

We'll focus on floats in [0,1]

8 bits are used on disk and for fast computation but dealing with 8 bit arithmetic is a pain. We discuss it in later chapter (ref).

• side bar: **how floating point works** — a float stores a number as **sign × mantissa × 2^exponent**, binary scientific notation: the **exponent** slides the binary point so the same ~24 mantissa bits (32-bit `float`) give **relative** precision across a huge range. What this means for images:

• **precision is relative** — fine resolution near 0, coarser for large values; a good match for **linear radiance / HDR** (and roughly for how we perceive light)

• values **outside [0,1] are fine** (super-whites, the negative lobes of a sharpening filter) — nothing clamps until you output

• `0.1` is **not exact** in binary, and adding numbers of very different magnitude **loses the low bits** (a long sum of many small values drifts) — usually negligible for images, occasionally not

• **inf** and **NaN** (`0/0`, `log` of a negative, `0×inf`) **propagate** and silently poison an image — a classic bug source; check for them

• `float32` is the working default; **`float16`/half** halves memory (ML, some HDR pipelines), `float64` is rarely needed for images

• [forward ref: the pains of **8-bit integer** arithmetic — overflow, rounding, no headroom — later chapter]

• **reminder** — 💡 **L2.49, quantization is rarely the real problem** (the lesson is introduced in FUNDAMENTALS → [[Image Formation and linear perspective#Noise|Noise]]): with enough bits and a sane encoding it's **noise and dynamic range** that bite, not the number of levels — which is exactly why **floats with headroom** are the sane default here.

• **how many bits for what** (match the depth to the *job*):

• **8-bit integer** — **display and delivery**, and compact storage. ~256 levels is enough *perceptually* **only because gamma** puts the codes where the eye is sensitive (the darks — see *encoding* below); but it has **no headroom** and **rounds on every operation**, so it's a poor *working* format.

• **10–14-bit** — what **camera raw** delivers, sized to the sensor's dynamic range (and to dual-conversion-gain / HDR sensors, FUNDAMENTALS).

• **16-bit integer** — the safe **intermediate for editing**: repeated operations don't accumulate visible **banding / rounding** the way 8-bit does (Photoshop's "16-bit mode").

• **float32 (linear light)** — the sane **working / HDR** format: relative precision, values allowed **outside [0,1]**, nothing clamps until output. **float16 / half** halves memory for ML and HDR pipelines.

• **the rule of thumb**: **more bits — and float — wherever you do *math* or hold *high dynamic range*; few bits only at the display / output end.** Spend precision where rounding and range actually hurt.

• 💡 **bit-depth and encoding decisions go together.** How many bits you need depends on the **encoding** you store them in, because a big part of **gamma's job is to give you better *uniform* quantization**: it spaces the available codes the way the eye perceives brightness — fine steps in the darks, coarse in the highlights — so **8 *gamma*-encoded bits** look smooth where **8 *linear* bits** would band badly in the shadows. (That is not gamma's *only* job — it also matches display response and historical pipelines, see *encoding* just below — but "make a few bits go further" is a big one.) The converse: you do **math in linear**, and linear light wants **more bits / float** to avoid that banding. So don't pick a bit depth and an encoding separately — **choose them as a pair**.

• 💡 **Big lesson (L2.52 · dither: trade structured error for unstructured noise).** When too few levels force rounding, the error is **correlated** → a smooth gradient breaks into **bands** the eye locks onto. **Dithering** spends that error as **high-frequency (blue) noise** instead so the local average tracks the true tone — the eye **integrates noise but sees structure**. [`fig-dithering-methods`]

• **stochastic quantization first** (the simplest form): round each pixel up/down **at random** with probability set by its position between levels → expected value exact, bands → white noise; this is **stochastic rounding** (unbiased low-precision NN training). No memory → noisier than needed.

• **error diffusion** gives the rounding *memory*: Floyd–Steinberg [@floyd-steinberg-1976] pushes each pixel's residual onto not-yet-visited neighbours → far less noise; **Ostromoukhov** [@ostromoukhov-2001] (intensity-dependent serpentine coefficients + a touch of blue noise → cleanest). Cleanest at **1 bit** (no interior levels to seam).

• ⚠️ **diffuse the error in LINEAR light**, not code values — the eye averages light, so only then does the dither's average brightness match the original (gamma/code-space dithering lifts the mid-tones; Jensen on the display curve).

• **not just for the display:** the same scatter-the-error trick recurs in **coded imaging** (binary/few-level masks), **computational illumination**, blue-noise sampling, and **stochastic rounding** in low-precision arithmetic. Registers **L2.52** here.

say clearly when things are encoded, when things are decoded.

KEY framing: a bare array of numbers is meaningless without knowing its encoding / color space

Discuss that different aspects of comp photo call for different encoding: linear (e.g. for dealing with physics, white balance, radiometry) vs. gamma vs. log

• rehash gamma and other encoding, ref to early section/chapter

• JPEG/film are usually some weird remapped versions for beautification

• as a result, you have more radiometric approaches to image editing/processing (starting with calibrated/linear/RAW values), vs older approaches that are more hand wavy because values were less clear

• the same three letters hide real differences: **channel order** (RGB vs BGR — OpenCV), **color space / primaries** (sRGB vs Adobe RGB vs Display P3 → Color technology), **encoding** (linear vs gamma), **range** (`[0,1]` float vs `[0,255]` uint8 vs 10/12-bit), and **premultiplied vs straight alpha**. A bare array doesn't say which — read the metadata / profile, don't assume.

• an image carries more than the pixel array: **basic metadata** — pixel **width** and **height**, channel count, bit depth, the file **name / path**, creation date

• richer **EXIF** capture metadata written by the camera: **ISO, shutter speed, aperture, focal length**, lens, timestamp, **GPS**, **orientation** flag, embedded ICC profile

• ⚠️ **EXIF is frustrating in practice**: not every camera writes every field and the set / format is **inconsistent** across makers; some values (notably **ISO** and **shutter speed**) are **approximate / rounded** and **not radiometrically correct** — don't treat them as calibrated measurements

• **concrete numbers** (so the warning isn't hand-waving): the recorded **shutter speed is the *nominal* one** — a phone that actually exposed for 1/3.0 s or 1/3.9 s may write **"1/3 s" in both cases**, a photometric error of up to **~30 %** (the kind of EXIF exposure-time rounding documented in consumer-camera calibration work, [@burggraaff-etal-2019]); and the **EXIF ISO is the manufacturer's *setting*, not the measured sensitivity** — independent RAW measurements (DxOMark, ISO 12232 saturation method) routinely find the **real ISO differs from the marked one by up to ~⅓–1 stop** (a body set to "ISO 100" may measure ~80, "ISO 3200" may measure ~2000), because makers tune the gain/headroom and the standard leaves slack. So for any **radiometric** use (HDR weights, noise modelling, exposure fusion) treat EXIF shutter/ISO as a *hint*, and prefer a per-body **calibration** or the **DNG `NoiseProfile`** when available. (Refs: [@burggraaff-etal-2019]; DxOMark *ISO speed* glossary + sensor protocol; ISO 12232.)

• a catalog of the typical fields lives in the appendix → [[Common EXIF fields]]; the file-format side of metadata is in [[File formats and compression#Data versus metadata: EXIF|File formats]]

• 💡 **what metadata *should* travel with an image** (a recurring wish in this book): beyond the basics, a sane representation would carry exactly the things a bare array can't imply — the **encoding** (linear / gamma / log), the **color space / primaries** (ideally an embedded **ICC** profile), the **noise level** (the read + shot noise parameters, so downstream code knows the per-pixel uncertainty), and — when known — the **point-spread function (PSF)** (for deconvolution / deblurring). Most formats carry the first two unreliably and the last two not at all, yet later algorithms (white balance, **denoising**, **deblurring**, HDR) silently *need* this — so an honest image is pixels **plus** this provenance.

• 💡 **noise level deserves to be metadata — and almost never is.** Of that wish-list, the **noise level** is the most useful and the most neglected: a tiny per-sensor calibration (read noise + a shot-noise/gain term, i.e. the variance as a function of signal at the current ISO) tells downstream code **how much to trust each pixel**. It is **rarely written into any image file**, yet **denoising, HDR merging, deblurring, and any weighted least-squares fit** all silently need it — without it they fall back to guessing the noise from the image. It costs only a couple of numbers to store, so carrying it alongside the pixels would make a lot of estimation principled instead of ad hoc. (**DNG**'s `NoiseProfile` tag is a rare format that actually does this.) → [[Noise, signal-to-noise ratio and dynamic range]], [[Denoising basics]].

• preview the taxonomy of image operations by what they touch (point / domain / neighborhood) — developed in Point operations

packing multidimensional arrays in memory. Various options.

Which one we choose (different in Python and our C++ implementation)

• channel order: RGB vs BGR (e.g. OpenCV)

• interleaved (HWC) vs planar (CHW); CHW is what PyTorch wants later

• **the C++ image data structure** (C++ profile — show this as the book's working representation): a minimal `Image` holding `width, height, channels` and a flat `std::vector<float> data` in **planar** order, indexed `data[c*width*height + y*width + x]`, with a `float& operator()(int x, int y, int c)` accessor hiding the index arithmetic. (Python profile: a NumPy `H×W×C` array.)

origin top-left (y down) vs bottom-left (y up)

pixel center vs pixel corner: the half-pixel offset — a classic bug source; matters for resampling / warping (where-from vs where-to)

• **the problem**: a `float[H][W][3]` is **mute** — it does not say whether it is **linear or gamma**, **sRGB or P3**, `[0,1]` or `[0,255]`, **straight or premultiplied** alpha. Nearly every "why are my colors wrong?" bug is an operation applied to data in the **wrong type** (e.g. white-balancing or blending gamma-encoded values as if they were linear). The cure is to make the image **typed**.

• **a good image type encodes its gamma (and the rest of its provenance)**: alongside the array, a small **type record / tags** — **encoding** (linear / gamma / log), **color space / primaries** (ideally an embedded **ICC** profile), **range and bit-depth**, **alpha kind** (straight vs premultiplied). The encoding is **part of the value, not a fact you keep in your head**. (This is the "what metadata *should* travel with an image" wish from *Beyond the pixels*, promoted into the type.)

• **conversion = cast = convert the numbers AND re-tag** (the key discipline): a `to_linear()` must **divide out the gamma *and* set the encoding field to `linear`**; a `to_uint8()` must **scale and round *and* set the range / bit-depth tag**; a color-space convert applies the 3×3 *and* updates the **primaries** tag; (un)premultiply flips the **alpha** tag. A cast that changes the numbers **without** updating the tag (or the reverse) is exactly the bug. Done right, **type and data are always consistent**, and an operation can **refuse, or silently auto-convert,** a mismatched input.

• **why it's worth the ceremony**: it makes the **illegal states unrepresentable**. White balance and exposure can *insist* on **linear** input (multiply-in-linear, [[Measuring and encoding color|Color technology]]); a blend can *insist* on matching **alpha kind** and **color space**; a save can *insist* on an output **encoding**. A type check catches the classic mistakes up front, instead of the *viewer* catching them later. It is **units for images**.

• **in practice**: most arrays are untyped (NumPy, OpenCV) — which is *why* these bugs are everywhere; but **color-managed** apps, Adobe's pipeline, and **DNG-style** containers ([[Appendices#DNG: the Digital Negative]]) effectively carry this type, and the book's own `imageops` treats encoding/space explicitly. The takeaway: **track the type, and let conversions move it — don't keep it in your head.**

• **RGBA**: a 4th **alpha** channel = per-pixel opacity / coverage; **premultiplied** (RGB already scaled by α) vs **straight** alpha is a classic bug — full treatment in **Compositing**

• other extra channels: a depth / Z channel, a matte, or **hyperspectral** (many narrow bands) — the array just gains channels

• **video** = an image sequence: just an extra **time** axis (H×W×C×T); most per-frame operations generalize directly (and motion couples frames — Motion/Video)

• **higher-dimensional** imaging: volumetric **3D** (medical CT/MRI: H×W×D), **4D light fields** (two spatial + two angular), spectral cubes — the same array machinery, more axes

• wrap pixel access in a **safe accessor** so out-of-bounds reads don't crash or read garbage — and decide what a pixel *past the boundary* means:

• **black / zero** (pad with 0 — darkens edges), **clamp / replicate** (repeat the edge pixel), **mirror / reflect** (fold the image back), **wrap / tile** (periodic — what the DFT assumes)

• the choice matters for **convolution, resampling and warping** at borders; mirror is the usual safe default [edge-handling modes figure]

• your favorite IDE or text editor; build, run, repeat

• the key habit: **save intermediate images to disk** — the "printf" of image code

• optional: display from code; a small UI (html?)

• 💡 **Big lesson — most debugging is tedious but simple, not clever.** **90% of debugging is methodical, not inspired**, and it comes down to two things: (1) **choose smart inputs** — small, hand-verifiable cases where you already know the right answer (constant, impulse, edge — see *Test on inputs you can verify by hand*, below); and (2) **understand the execution** — walk through it **step by step** and **output / print as many intermediate values and images as you can**. The bug almost always reveals itself the moment you *look* at an intermediate, instead of reasoning about it in your head. The rest of this section is just that lesson, unpacked.

• doubt everything; assume no piece works until you've checked it

• test incrementally — never write the whole program before testing a piece

• isolate pieces; binary-search / divide and conquer to localize a bug

• change one thing at a time

• display, don't guess: dump intermediate results instead of reasoning in your head

• to confirm a step does what you think, deliberately break it and watch the output change

• small (e.g. 3×3) synthetic images: constant, impulse, edge / half-plane (half black, half white), rectangle

• feed these to intermediate stages too, not just the final output

• each algorithm also has its *own* hand-checkable test inputs — those recipes now live as a **debug sidebar in each algorithm's own chapter**: impulse → kernel in [[Neighborhood operations and convolution]]; constant / rectangle in [[Demosaicking]]; the $\sigma_r$ limits of the **bilateral filter** in [[Bilateral filtering]]; a 1–2 px shift for **image alignment** in [[Multiple exposure imaging|Multiple-exposure imaging]]

• a seg fault usually surfaces far from its cause; suspect an array/vector index

• print the index and the array size; **assert index ∈ [0, size)** in a debug build — ties to the edge accessor in Image representation

• mostly good, not always predictable (e.g. antialiasing, bad cross-fading)

• **like all AI — sometimes remarkable, sometimes it fumbles something trivial**: reproducing **Kemelmacher-Shlizerman & Seitz's *Photobios*** ([[Many images and photo collections]]), the model autonomously nailed the *hard* parts — **face registration** from landmarks and a **dynamic-programming** path through the photo collection — yet repeatedly botched a **one-line cross-fade** `frame = (1−t)·start + t·end`, needing ~5 re-prompts to get it right. Lesson: **review even the trivial parts** — competence on the hard stuff is no guarantee on the easy stuff.

• **vibe debugging / vibe explaining**: use the LLM to explain an error, propose a minimal repro, or suggest where to look — but **verify against a hand-checkable input** (above) before trusting a fix; treat generated image code as *guilty until tested*

• our guidance: keep the LLM on a short leash for **numerically / perceptually subtle** code (filtering, resampling, color, antialiasing) where "looks plausible" ≠ correct; lean on it for boilerplate, I/O, and visualization

• 🔁 **quick reminder** (already previewed back in *Image representation → Three kinds of operation*): image operations split by **what they touch** — **point** (range; same pixel's *value* → one 1-D curve), **domain** (a *different* location → resampling/warping), **neighborhood** (a *window* → convolution/bilateral). No need to re-derive it here — this chapter just **zooms into the point branch**, the simplest, and where we start. [the three operation types figure, with the **point-operation** branch **highlighted**]

• the **histogram** (the distribution of values — its own chapter, next) is the natural companion to point ops

• develop everything on a grayscale value; for color, apply the same curve per channel (saturation is the exception)

for single channel : simple 1D input output curve. the whole operation *is* that curve (the editor's curves tool); identity = the 45° diagonal

one big question: in which units (linear, gamma, log)

Say that traditional photography has always involved controlling the mapping from scene radiance to output photo tone.

side bar: the zone system. Zones are intensity ranges. encourages photographers to think/act on mapping from scene to negative and then from negative to print. Sometimes in a local way. Lots of tools: exposure, type of film, pre-exposure, chemistry, darkroom.

• 💡 **Big lesson:** **additive vs multiplicative → choice of encoding** — ask whether a tonal op is additive (→ linear/display offset) or multiplicative (→ linear light, a sum in log); light and perception are multiplicative

• exposure is multiplicative in linear space ($L_\text{out} = 2^s L_\text{in}$, one stop = a factor of 2)

• the +1-stop curve bows upward in display space; highlights clip at 1 (unrecoverable)

• do it in linear light; the naive gamma-space multiply clips at input ½

• brightness = additive offset $L_\text{out}=L_\text{in}+b$; lifts blacks, fogs the shadows

• recommend against brightness; exposure is more radiometric (keeps black at black). Brightness survives as a legacy slider / quick shadow lift

• contrast = steepen about a pivot $L_\text{out}=g(L_\text{in}-p)+p$; gain >1 steepens and clips both ends

• big question: in linear, gamma, or log (three legitimate looks; defer the full argument to tone mapping)

• sidebar: the S-curve (sigmoid) — roll off the extremes instead of clipping

• **levels**: set the **black point** and **white point** (+ a mid-grey **gamma**) — a piecewise remap that stretches the used range to [0,1]; the workhorse first edit

• read them off the **histogram**: clip the empty ends, pull the mids with the gamma slider

• a free-form **tone curve**: drag control points, smoothed by a **spline** kept **monotone** (Catmull-Rom / monotone cubic) so tones never invert [tone-curve / spline figure]

• **sidebar — splines (the math)**: fit a smooth curve through the control points (knots) $(x_i,y_i)$ as a **piecewise cubic** — one cubic per interval, the pieces joined for continuity.

• **cubic Hermite** form on $[x_i,x_{i+1}]$ with $t=\frac{x-x_i}{x_{i+1}-x_i}\in[0,1]$: $p(t)=h_{00}(t)\,y_i+h_{10}(t)\,m_i+h_{01}(t)\,y_{i+1}+h_{11}(t)\,m_{i+1}$, with basis $h_{00}=2t^3-3t^2+1,\ h_{10}=t^3-2t^2+t,\ h_{01}=-2t^3+3t^2,\ h_{11}=t^3-t^2$. The knot **tangents** $m_i$ are the free choice — picking them is what separates the schemes.

• **Catmull–Rom** (interpolating, $C^1$): tangent = centred difference $m_i=\frac{y_{i+1}-y_{i-1}}{x_{i+1}-x_{i-1}}$ — local and cheap, but can **overshoot** (so it may go non-monotone).

• **natural cubic** ($C^2$): require continuous second derivatives and minimize total curvature $\int (p'')^2$, which gives a **tridiagonal** linear system for the knot slopes — smoothest, but still can overshoot.

• **monotone cubic** (Fritsch–Carlson): start from Hermite tangents, then **clamp** them so every segment is monotone — if a secant slope $\Delta_i=\frac{y_{i+1}-y_i}{x_{i+1}-x_i}=0$ set the neighbouring tangents to $0$; otherwise keep $(\alpha,\beta)=(m_i/\Delta_i,\ m_{i+1}/\Delta_i)$ inside the circle $\alpha^2+\beta^2\le 9$. This **guarantees no inversion**, which a **tone curve** requires (brighter in $\Rightarrow$ brighter out). [tone-curve / spline figure]

• **B-splines** are the *approximating* cousin — the curve follows a control polygon and need **not** pass through the points; smoother and easy to keep monotone, at the cost of exact interpolation.

• subsumes exposure / contrast / levels; container for the principled tone-mapping curves (next chapter)

• **sidebar — how analog photography shaped the same mapping**: long before a digital curve, photographers controlled the **scene-tone → printed-tone** mapping by physical and chemical means —

• **exposure**: where you *place* a scene luminance on the film's response (the Zone System's "place this shadow on Zone III, let the highlights fall where they may")

• the film's **characteristic (H&D) curve** and its **gamma / contrast** — chosen by emulsion *and* by **development** (time / temperature / agitation / developer; **push / pull**; "**N / N+1 / N−1**" development to expand or compress the scene's range onto the film)

• **pre-exposure / flashing**: a uniform sub-threshold exposure that lifts deep shadows above the film's *toe* (raising the floor without touching highlights)

• at print time: the **paper grade** (contrast grade 0–5), the print developer, and local **dodging & burning** (hold back / add light per region)

• the free-form tone curve (and the per-region masks of *Local point operations*, below) is the **digital descendant** of this whole toolkit. *Cite **Ansel Adams**' trilogy — **The Camera**, **The Negative**, **The Print** — and the **Zone System** (developed in *The Negative*).*

• the **basic color edits** every editor exposes — the color analogue of the tone curve. **This is the home for the implementable mechanics** (the equations below); the *perceptual-space* nuance — which space to scale chroma in, why skin needs protecting — is the job of [[Measuring and encoding color|Color technology]] (don't defer the equation there)

• **saturation**: split each pixel into **luminance** $Y$ and **chroma** $(\text{RGB}-Y)$, scale the chroma, recombine — $\text{RGB}' = Y + s\,(\text{RGB}-Y)$ (so $s=0$→grey, $s=1$→identity, $s>1$→more saturated; equivalently $(1-s)\,Y + s\,\text{RGB}$). A *uniform* $s$ over-saturates already-vivid colors and skin

• **vibrance** (implementable): the same operation but with a **per-pixel gain** that fades as the pixel's own saturation rises, and protects skin — replace the constant $s$ by

$$g = 1 + v\,(1-\text{sat})\,w_\text{skin}(\text{hue}), \qquad \text{RGB}' = Y + g\,(\text{RGB}-Y)$$

where **sat** is the pixel's current saturation (e.g. HSV $S = \tfrac{\max-\min}{\max}$, or chroma magnitude / luma), **v** the vibrance amount, and **w_skin(hue)** a hue weight $\approx 0$ on skin / orange hues and $1$ elsewhere — so muted colors get the full boost while vivid colors and skin barely move (set $w_\text{skin}\equiv 1$ for the skin-agnostic version)

• which space to scale chroma in (perceptually uniform), and why skin needs protecting — deferred to [[Measuring and encoding color|Color technology]]

• **LUT (look-up table)**: bake any color/tone mapping into a precomputed table indexed by pixel value, so applying it is one **lookup per pixel** — fast, and the standard way to ship a *look*

• **1-D LUT** = three independent per-channel curves (exposure / contrast / levels / gamma collapse into one); **3-D LUT** = a full RGB→RGB map that handles **cross-channel** effects a 1-D LUT can't (white balance, film/grade looks — the `.cube` files of video), at the cost of a coarser sampled grid + interpolation

• LUTs are also a **speed** trick: precompute an expensive per-value function once, then index it (ties to fixed-point / 8-bit pipelines)

• a **separate craft** from stills / radiometric editing — the *look* of motion pictures, with its own tools and vocabulary; less radiometric, more color craft

• the colorist's primaries: **lift / gamma / gain** wheels (shadows / midtones / highlights, each with a color offset); **primary** (whole-frame) vs **secondary** (qualified by hue / luma, or masked) corrections

• the working space is usually a camera **log** encoding (S-Log, Log-C) under a color-management pipeline (**ACES**); the finished *look* is shipped as a **3-D LUT** (`.cube`), applied per frame (ties to [[#Lookup tables]])

• the **teal-and-orange** look: push skin into warm **orange**, everything else toward its **teal** complement — complementary contrast that makes people pop; popularized by ***O Brother, Where Art Thou?*** (2000 — among the first features with a full **digital intermediate**; Coen brothers, cinematographer Roger Deakins)

• the **stills-side cousin**: Lightroom's HSL / color-mixer and color-grading panels

• ties: [[#Lookup tables]] (3-D LUTs are the delivery format); [[Measuring and encoding color|Color technology]] (color management, log / ACES); tone mapping (the dynamic-range side)

• strictly not a point operation: apply a tool through a **selection mask** (the result depends on *where*, not just the pixel's value)

• **selections**: by value / color (magic wand), by **gradient** (linear / radial), by **brush**, **edge-aware** (snap to edges → Big lesson: edge-preserving = affinity), and increasingly **AI / semantic** (segment-anything-style)

• 💡 **the moment a point operation goes *local*, it raises the central question of "*which pixels?*" — segmentation and matting.** A hard binary **selection** is a *segmentation*; a soft-edged one (feathering, hair, motion blur, partial coverage) is a **matte** (a per-pixel α, exactly the alpha channel of [[#Alpha and extra channels]]). So even a humble masked exposure tweak is the front door to a deep topic — how to decide the mask, and how to make its border *soft and correct*. **Forward ref → [[Compositing, segmentation and matting]]** (the full treatment: matting equation, trimaps, alpha estimation), with edge-aware mask refinement in [[Guided image filtering]] / [[Bilateral filtering]].

• the bridge to **local tone mapping** and edge-preserving filtering (next chapters)

• the **histogram** = the distribution of pixel values; reading it tells you exposure, clipping, contrast, and where the tones sit [histogram figure]

• the on-screen **RGB and/or luminance histogram** in the camera and in editing apps; a **live histogram** in the viewfinder / on the rear screen while composing

• **clipping warnings** ("**blinkies**"): blown highlights and crushed shadows flash to flag lost detail at each end

• ⚠️ the displayed histogram is typically of the **gamma / display-encoded** image, **not** the linear data — so its shape (and where mid-grey sits) reflects the encoding, not scene radiance (ties to *depends on the encoding space*, below)

• beyond per-channel: a **3-D color histogram** = the distribution of pixels in **color space** (a cloud / occupancy in the color volume)

• its shape **depends on the color space** you build it in — RGB cube vs a perceptual space (Lab / opponent) bin the same pixels very differently

• used for **palette extraction, color transfer**, and **segmentation** (clusters in color space)

• its **shape depends on the encoding space** — the *same* image has a very different histogram in linear vs gamma vs log [histogram-encoding-spaces figure: linear piles into shadows, gamma spreads it, log spreads shadows more; 18% gray lands at 0.18 / ~0.46 / ~0.75]

• warning: it's a **sampled / discretized** estimate — spikes and gaps are artifacts of **binning / quantization**, not the scene

• **histogram equalization**: use the image's **cumulative histogram (CDF)** as the tone curve → flattens the histogram, maximizing global contrast (often too aggressively) [histogram-equalization figure: the CDF *is* the transfer curve]

• pseudocode: bin → running-total CDF scan → normalize → apply as a per-pixel LUT

• **histogram matching** (a.k.a. histogram **specification**) generalizes equalization: instead of flattening to a *uniform* target, reshape an image's histogram to match **any target distribution** — a chosen shape, or *another image's* histogram

• **the recipe = compose two CDFs**: equalize the source (push it through its own CDF, $\text{CDF}_\text{src}$, to a uniform), then **"un-equalize" through the target's inverse CDF** — $L_\text{out} = \text{CDF}_\text{tgt}^{-1}\big(\text{CDF}_\text{src}(L_\text{in})\big)$; both are 1-D LUTs, so matching is still a single **point operation** [histogram-matching figure: source histogram + target histogram → matched result + the composed transfer curve]

• **equalization is the special case** where the target is uniform ($\text{CDF}_\text{tgt}=$ identity) — so matching, equalization, and a hand-drawn levels curve are one family

• **for color**: matching each channel independently **shifts hues** (the channels aren't independent), so do it in a **decorrelated / perceptual** space, or use the proper multi-dimensional version — **color transfer** (Reinhard et al. 2001: match the **mean and standard deviation** per channel in the decorrelated $l\alpha\beta$ space), with the exact N-D form via **optimal transport** (Pitié et al.)

• 💡 forward ref: histogram / color matching is the workhorse behind **harmonization** (make a pasted region's tones match the destination — Compositing), **color grading / style transfer** (impose a reference look), and **texture synthesis** (match value statistics) — all instances of "make this image's statistics look like that one's"

• refs: Reinhard, Ashikhmin, Gooch & Shirley 2001 (color transfer between images); Pitié, Kokaram & Dahyot 2007 (automated N-D color transfer / optimal transport); Gonzalez & Woods (histogram specification — textbook)

• **Greg Ward's constrained / histogram-based tone mapping** (1997): equalize, but **cap the local slope** so contrast is never exaggerated beyond what the eye accepts — a perceptually-bounded version for HDR [Ward et al. 1997]

• **dynamic range** = ratio of brightest to darkest (contrast, not exposure); scene ~1:100,000, sensor ~1:1000, display ~1:20–500

• two problems: **capture** (merge multiple exposures → HDR chapter) vs **display** (squeeze a huge range onto the screen *while keeping detail* — this chapter); can't just rescale (shadows collapse)

• sidebar: limited range can be a virtue (W. Eugene Smith; lighting reduces range at capture)

• the two families map exactly onto the **global vs local adjustments** split that organizes all editing (point operations = *global* adjustments; masked / per-region tools = *local* adjustments — see *Point operations* and *Local point operations*):

• **global tone mapping** = **one curve everywhere**, which is precisely a **point operation / global adjustment**. It is *already in the point-operation toolbox* (the Reinhard / log / S-curve / freehand tone curve of *The general tone curve*), now aimed at range compression — simple, fast, no halos, but a flattened slope loses local contrast. **So global tone mapping files with the global adjustments — mechanically nothing new, just a particular choice of the global curve.**

• **local tone mapping** = the curve **varies spatially** (a **neighborhood operation / local adjustment**): split large-scale vs detail, compress only the large scale → needs an **edge-aware blur**. This is the genuinely new machinery, and why local tone mapping is treated *after* point operations and leans on the edge-preserving filtering of *Edges matter*. [global-vs-local figure]

• *(this is the global-adjustment / point-operation member of the pair above — kept here only as the HDR application of the tone curve already built under [[#Point operations]])*

• naive clip / rescale both fail (linear answer to a multiplicative problem)

• the **Reinhard** global operator `L_out = L/(1+L)` (or a white-point variant) maps `[0,∞) → [0,1)` — a principled, halo-free default global tone map; it is **one entry in the global tone-curve family** (gamma, log, S-curve, freehand) — the same 1-D point-operation curve, chosen for range compression [Reinhard curve figure]

• **one curve for the whole image** — simple, no halos, but can't squeeze a huge range into a small one without flattening local contrast

• apply to **luminance** only, rescale RGB by the ratio (keep hue)

• **histogram equalization** is itself a tone map: CDF as the curve, data-driven, max global contrast — but no restraint (exaggerates busy regions, amplifies noise)

• **Ward et al. 1997**: cap the local slope → perceptually-bounded equalization for HDR

• *(this is the **local-adjustment** member — a neighborhood operation, so it sits **after** the point-operation / global-adjustment material above and depends on machinery built later in **convolution** and **edge-preserving filtering**)*

• the curve **varies spatially** (compress the slow, large-scale variations, keep local detail) — needs a **blur / edge-aware** step, so it waits for convolution & edge-preserving; its failure mode is **halos** around strong edges [local-tone-mapping halo figure]

• best-of-both decomposition: large-scale layer + detail layer; compress only large-scale, add detail back

• naive Gaussian blur crosses edges → contaminated large-scale layer → **halo** band at strong edges; fix = edge-preserving (bilateral) blur

• 💡 **Big lesson:** **edge-preserving = affinity** — weight a blur by value affinity as well as spatial distance (the bilateral filter)

• **log encoding pays off here**: range compression is multiplicative, so log turns it into a simple subtraction (→ Big lesson: additive vs multiplicative)

• detail = log(L) − large-scale (a subtraction); compress + recombine = additions

• 💡 **Big lesson:** **additive vs multiplicative → work in log** — light/exposure/range are multiplicative; log turns them into addition, which blurs/decomposes/compresses cleanly

• Ansel Adams' **Zone System** = global tone mapping: carve light into 11 zones (a stop apart), Zone V = 18% grey; place a scene tone in a zone, realize it through exposure/development/paper — the analog of Reinhard's white point [Zone System strip figure]

• side bar: **dodging & burning** — the darkroom ancestor of local tone mapping (hold back / add light locally), even local contrast / sharpness

• **video grading evolved separately** — less radiometric (lift/gamma/gain), rich craft; same operations, different dialect → see [[#Video color grading]] (Point operations)

• ⚠️ honest note: **full HDR tone mapping is not very commonly used in practice** — it's **hard to control** and tends to look **artificial / "HDR-look"** (haloes, flat, garish)

• most photographers prefer **simpler exposure / contrast adjustments** and **exposure fusion** over an aggressive HDR operator

• recap: tone mapping = display-side range compression; global (halo-free, loses local contrast) vs local (crisp detail, needs edge-preserving blur, risks halos); all cleanest in log; HDR chapter picks up the threads

• a radiometrically-correct scene-linear render looks **dull and ~1–1.6 stops dark**; nothing is broken — it just lacks the camera/raw "look"

• KEY: do processing in scene-linear; apply the **beauty curve only at the very end, for display**; never bake it into data you compute on

• mid-grey is ~0.18 in linear, not 0.5 → honest encode lands midtones low; and a bare encode has no highlight shoulder

• the missing midtone lift measured vs a Lightroom JPG ≈ 1–1.6 stops

• tone mapping compresses an enormous range and is rare in practice; the beauty curve renders **every** standard-range photo; shapes rhyme, jobs differ

• smooth, monotone, bows above the diagonal (midtone lift) with a soft highlight shoulder; must be smooth (monotone cubic) — kinks/plateaus band the highlights

• real shape lives in the camera profile (Adobe baseline tone curve + HueSatMap + LookTable); we recover an equivalent baseline curve empirically (fit linear-render luma → developed-JPG luma)

• hue-preserving (scale all channels by the luma ratio) keeps highlights colored → magenta/cyan sky near the clip

• fix: **highlight-weighted hybrid** — hue-preserving in shadows/mids (keeps deep-blue sky saturated), blend to **per-channel** near white so highlights bleach cleanly to white; ease saturation off in the highlights too

• a gentle (~+10%) luma-preserving saturation finishes it; HueSatMap/LookTable are *style*, not necessity, and easy to get wrong (purple blues, garish greens) — stop at baseline curve + highlight-aware saturation

• process in linear; beautify last — Poisson/HDR/WB/deblur must see scene-linear, not the look-curved render

• a pixel computed from its neighbors (recall the three operation types)

• blur also comes from optics: diffraction, spherical aberration → motivates removing optical artifacts (forward ref: deblurring) so important to understand if we want to prevent and remove it.

• blur as a weighted average of neighbors.

• linear shift-invariant (LSI) filtering; why linearity matters (with a non-shift-invariant counter-example)

• the convolution algorithm: loop over the output

• point spread function (PSF) / impulse response

• **debugging / hand-checkable inputs**: an **impulse** reads off the kernel (the PSF); a **constant** must pass through a normalized kernel unchanged; a **shift kernel** just translates the image; a **half-plane** (edge) shows the edge response; verify a **1-D 3×3** by hand first

• without the flip it's called correlation. USeful too but not as useful and doesn't have all the good mathematical symmetries.

• sidebar: convolution in probability (sum of random variables, Laplace first derivation of convolution, flip is obvious: if we want the sum to be a given number, then the second number has to be the sum minus the first one)

• **sidebar — debugging convolution** (recipe moved here from Developing, testing & debugging): convolve an **impulse** and the output **is your kernel**, laid out and flipped — read it off number-for-number to catch a wrong weight, an off-by-one in centering, and the **flip** (convolution vs correlation) at a glance; a **constant** comes back unchanged iff the weights sum to 1; an off-centre single-1 **shift kernel** translates the image by exactly that offset (tests sign & direction); a **half-plane** shows the border behaviour; do the very first test in **1-D (1×3)**, where every output is hand-computable. [fig-psf-impulse]

• **normalization**: the weights sum to 1 to preserve brightness (a blur shouldn't darken the image)

• **commutativity / associativity**: I*g = g*I and (I*g)*h = I*(g*h) — so filters can be pre-combined and reordered (the flip is what buys these symmetries)

• **linearity**: scales and superposes — the very reason Fourier will help (next chapter)

• the **identity** is the impulse δ (I*δ = I); **separability** gets its own point below

• edge padding (ties to the edge/accessor policies above)

• centering the kernel

• box filter

• Gaussian (formula, truncation at ~3σ)

• a 2-D filter is **separable** if it's an outer product of two 1-D filters (the Gaussian is): blur **rows, then columns**

• big **cost** win: a (2r+1)×(2r+1) kernel costs O(r²) per pixel; done separably, **O(2r)** — the practical reason large Gaussian blurs are fast [separability figure]

• **derivative filters**: finite differences estimate ∂I/∂x and ∂I/∂y; the **Sobel** kernel pairs a derivative (one axis) with a little smoothing (the other) for noise robustness

• the **gradient** ∇I = (Iₓ, I_y) → its **magnitude** (edge strength) and **orientation** (edge direction): the basis of edge detection (forward ref: features / Canny) [gradient / Sobel figure]

• **the median filter — the simplest neighborhood op that *isn't* a convolution** (the first step off the linear path): replace each pixel by the **median** of its window. It is a **rank / order-statistic** filter — not a weighted average at all, so it can't be written as a convolution and has no frequency response. Two consequences make it the canonical nonlinear neighborhood op: (1) an **outlier can't sway it** — one wild value (salt-and-pepper / impulse noise, a hot pixel) sits at the *end* of the sorted window, far from the median → it's **rejected**, where a mean/box-blur would smear it into a grey blob; (2) it is **edge-preserving** — across a clean step, the window's median lands on whichever side holds the majority, so the edge stays sharp instead of ramping. It does round corners and erase thin lines, and it's the same robust-statistic idea as the temporal median (deghosting) and sigma-clipping — *use the consensus, not the average*. (Full denoising treatment: [[#Denoising basics]] → *Spatial averaging and its limits*.)

• a Gaussian blur **smears edges** because it weights neighbours by spatial distance alone (a near-but-different pixel pollutes the average)

• fix: weight each neighbour by **value similarity** too → the **bilateral filter** (Tomasi & Manduchi 1998): spatial Gaussian × range Gaussian on the intensity difference [edge-preserving figure]

• per-pixel normalization (weights depend on the values); **nonlinear** and **not shift-invariant** — leaves the clean world of convolution (no commutativity, no Fourier diagonalization)

• 💡 **Big lesson:** **edge-preserving = affinity** — use the color/intensity difference as how much two pixels "belong together"; the engine of edge-aware denoising/sharpening, halo-free local tone mapping, matting, colorization

• two kinds of weighted average: fixed weights → convolution (linear, shift-invariant); image-dependent weights → edge-preserving (bilateral, nonlinear)

• two threads forward: **Fourier** (next — convolution becomes per-frequency multiplication, enables deblurring) and the **affinity** lesson (recurs all book)

• unsharp mask: detail $=\text{in}-\text{blur}$, add it back amplified; flat regions untouched, edges/texture boosted; the whole operation is one $\delta-g$ convolution (centre-positive, surround-negative). [`fig-unsharp-mask`, `fig-sharpen-kernel`]

• the darkroom relic: sandwiching a negative with a slightly **blurred, low-contrast positive** ("unsharp mask") boosts edge contrast where the two disagree — exactly $\text{in}-\text{blur}$, done in silver. (Moved here from the in-line sidebar; now its own short section, between *Linear sharpening* and *Non-Linear Sharpening*.)

• the unsharp mask is a weighted **difference** (in − blur), and a blur is an **average**, so the result depends on the **space** you compute it in — same as tone mapping / color. Apply the same mask in **gamma** (stored values, the editor default), **linear** (physical light), or **log** (ratios) and you get three different looks, most visible in the **edge halos**: linear → strong bright halo / weak dark; gamma → perceptually even; log → gentle in highlights, firm in shadows. No single right answer; worth seeing. [`fig-sharpen-space`]

• linear unsharp masking has two failure modes: it **amplifies noise** (in flat areas the high frequencies are mostly noise) and **haloes** at strong edges. [`fig-oversharpen-halo`]

• **adaptive / non-linear** unsharp masking **gates** the boost — sharpen **more in detailed / edge areas, less in smooth ones** (by local **edgeness** or **intensity**); cubic / rational variants (Ramponi) and adaptive control (Polesel et al.); apply mostly to **luminance**, in **midtones**, near **edges** — the principled version is the edge-preserving part. [`fig-sharpen-demo`]

• forward-ref: pushed past *amplifying* surviving detail into *inventing* it is **super-resolution** ([[Super-resolution]]); the edge-aware "gate by who-belongs-together" is the **affinity** idea developed in [[Bilateral filtering]].

Why? Allows us to use linear algebra, e.g. to solve and understand inverse problems such as deblurring.

cite Strang.

Useful both conceptually (translation to the language of linear algebra) as well as practically (use linear algebra libraries and well-established solvers)

Similar to how we store in memory: serialize all pixels/channels.

Note: one source of notation overload: images are often named x.

Operations on vector spaces and what they mean for images: addition, multiplication by scalar

dot product aka scalar product aka dot product aka inner product

Matrices without forming matrices

Once images are modeled as vector, linear operators (exposure, convolution) can be represented as matrices. But giant matrices. In a case like convolution

And just like with any vector space we can do a change of basis. See next.

Convolution is a linear operation. It can be represented as a matrix. But matrices and linear operations can be unwieldy to understand. For example, we may want to know how easy it it to invert a matrix (if we want to deblur).

Said differently, the output of a convolution for a pixel depends on other pixels and these cross-dependencies create complexity.

However, there is a special case of matrix that is easy to manage : a diagonal matrix. In a diagonal matrix, there is no cross-talk, each input just undergoes a scalar multiplication, which means the inverse is a simple scalar division. The difficulty of inversion boils down to how close the scalar is from zero.

Fortunately for us, we can do a change of basis (change of image representation) so that convolution turns into a diagonal operation. That change of basis is the Fourier transform. Warning: the only bug is it assumes that the image is cyclic. But still insightful.

And we have **already seen** that frequency is a good way to model **human vision**: the contrast sensitivity function (**CSF**) back in the perception chapter (Spatial vision) describes the eye's sensitivity as a function of *spatial frequency*. So the Fourier view is not only a computational convenience for convolution — it also matches how the visual system is organized, which is why frequency keeps coming back (JPEG, image-quality metrics, tone mapping). → see Human perception → Spatial vision / CSF.

• **sidebar — a Fourier transform you can see** (motivation: it's physical, not just an imposed trick): the little **starburst** a distant point of light (a star, a streetlight) makes through a small aperture/lens is ≈ the **Fourier transform of that aperture** (squared magnitude) — round hole → **Airy** disc + rings; stopped-down iris → **sunstar** spikes (one streak ⟂ each blade-edge). The optics performs the transform **for free**, in the instant light crosses the lens — no FFT. Lesson: Fourier is a *natural, physical* way waves + apertures organize themselves; we're borrowing it for images. → ties to [[10 Compound lenses, and aberration correction]] (diffraction / the **MTF** = FT of the PSF).

We focus on greyscale but it's the same for RGB.

Fourier transform: instead of one value per pixel, one coefficient per sine wave.

It's orthonormal, so coefficients are just dot products (sum/integrals)

• complex exponential carries a cosine + sine per frequency (Euler $e^{i\theta}=\cos\theta+i\sin\theta$); real part = cosine content, imag = sine — needed to capture any phase

• ⚠️ **so many Fourier conventions**: complex-exponential vs real sin/cos form; whether the $2\pi$ lives in the exponent, in the frequency, or in the normalization; continuous transform vs Fourier series vs DFT; normalization ($1/N$, $1/\sqrt N$, or none — and which side carries it); sign of the exponent ("forward vs backward"); angular vs ordinary frequency. These differ across textbooks, NumPy, and signal processing — **always check which convention a source uses** before you trust a formula. [fig-fourier-conventions — author's own "Fourier flavors" convention-warning illustrations, shown **full-width** so each reads clearly; from Fredo's slides 3_Linear_image_processing, slides 67–68 (author-created images)]

Figure: show change of basis matrix [⚠️ legibility: text labels currently unreadable — enlarge / redraw]

In general, Fourier diagonlizes all shift and shift-invariant operators.

• 💡 **Big lesson:** **diagonalize when you can** — express a linear operator in a basis where it's orthogonal/diagonal; **Fourier diagonalizes convolution**, turning it into per-frequency multiplication.

• **sidebar — Fourier and the heat equation** (where this trick came from): Fourier (1807) was solving **heat diffusion** in a 1-D rod with some initial temperature profile. His move was to write the temperature as a sum of **cosines** — and in that basis the heat equation **diagonalizes**: each frequency evolves on its own, and a single cosine simply **decays exponentially in time** (∝ e^{−k²t}), the highest frequencies fastest, so the profile smooths out. The cosines are the **eigenfunctions** of the Laplacian — the same reason Fourier diagonalizes convolution here.

• the heat equation also hands us the **two canonical solution families of linear differential equations: sine waves (in space) and exponentials (in time)** — separation of variables gives **spatial sinusoids** whose **amplitudes decay exponentially**, $\propto e^{-k^2 t}$, the highest frequencies dying fastest, so the profile smooths out. [fig-heat-diffusion-1d — a 1-D temperature profile (amplitude on y, position on x) smoothing over time: shown for (a) a general signal and (b) a single sine wave that simply decays in place]

• the general lesson: **diagonalizing is the win for any linear ODE / PDE** — in the eigenbasis a coupled system decouples into independent scalar equations whose solutions are **exponentials** (e^{λt}) or, for oscillatory / conservative systems, **sine waves**. Find the right basis (the eigenvectors) and a hard differential equation becomes trivial per-mode arithmetic.

• feed a pure sine into any blur → the **same sine**, only scaled and phase-shifted: sines are the **eigenvectors** of convolution, eigenvalue $\hat g(\omega)$ = the kernel's FT [sine-eigenvectors figure]

• blur leaves low freqs untouched, crushes high ones → blur is a **low-pass filter**

• the **convolution theorem** $\mathcal{F}\{I*g\}=\hat I\cdot\hat g$: convolution in space = multiplication in frequency

Give the example of a 2-pixel image visualized as a2D Euclidean plane. Our linear convolution has coefficients 0.8, 0.2 The Fourier transform is just along [1, 1] and [1, -1] (rotation by 45 degrees). Our simple convolution becomes diagonal

• 2-D: two frequency indices (u,v); display as **magnitude** + **phase**

• **symmetries**: a real image has a **conjugate-symmetric** spectrum (half the coefficients are redundant); low freqs at center, high at corners; natural images mostly low-freq; a shift → a **phase ramp**

• **magnitude vs. phase** — how to *read* an image's FT: the **magnitude** shows which frequencies / orientations are present; the **phase** carries the *structure*. Swap two images' magnitude and phase and you recognize the one whose **phase** you kept. [magnitude/phase figure]

• **compact in space ⇔ spread in frequency** — a narrow Gaussian has a wide spectrum; the locality / uncertainty tradeoff that recurs in pyramids & wavelets [compact-space-vs-frequency figure]

• sidebar: **MTF** for lenses: the optical system's frequency response is the FT of its PSF (ties to the diffraction limit, Optics)

• before we lean on it, two honest caveats that the rest of the book keeps bumping into:

• **the DFT treats the signal as cyclic / periodic**: it tiles the image and wraps around, so content leaking off the right edge re-enters on the left → **boundary leakage / ringing**. This is what motivates **windowing** before an FFT, and it is exactly the **periodic-boundary assumption** baked into the FFT deblur solver (forward ref: the FFT-based deconvolution in [[Computational tools]] inherits this wrap-around)

• **the basis has infinite / global support**: every coefficient is a **global wave** spanning the whole image, so there is **no spatial localization** — move a single pixel and *every* coefficient changes. A Fourier coefficient can tell you *which* frequencies are present but not *where*. This is precisely the motivation for **wavelets / pyramids**, which trade a little frequency precision for spatial locality → [[#Pyramids and wavelets]]

• the cyclic-boundary caveat is harmless when we use Fourier to *think* but a hazard when we *measure*: the wrap-around seam injects a spurious **"+"-shaped leakage cross** along the frequency axes that is not in the image. **Windowing** (multiply the patch by a Hann / Hamming / Blackman **taper** that falls to zero at the borders) kills the seam and the cross

• the **tradeoff**: a window suppresses side-lobe leakage but **broadens every spectral peak** (wider main lobe) and discards data near the borders — narrower window = less leakage but worse frequency resolution; Hann / Hamming / Blackman are just different **main-lobe-width vs side-lobe-height** points on that curve

• honest takeaway: we mostly use Fourier for **conceptual understanding** (leakage is harmless), but anyone who **measures** a spectrum — **MTF**, **noise power spectrum**, **periodic-pattern detection** — must window or mistake the leakage cross for real structure [fig-fourier-windowing — interactive: side-by-side 2-D FFT of a test patch (step edge / 2-D sine / sawtooth / box-pulse train) without vs. with a window; the leakage cross appears in the rectangular panel and vanishes in the windowed one, at the cost of broader peaks]

example of spatial aliasing. Show classical sine wave (prudential tower photo downsampling). Intuitively we need to break down the content that cannot be represented (return grey instead of black or white). You can think of it as the average over a pixel. More formally, we need Fourier.

As it turns out, sampling (and so aliasing) is **easy in Fourier — because Fourier is full of shifts.** Sampling is multiplication by a **Dirac comb** in space, which by the convolution theorem is a **convolution in Fourier**. That sounds harder, not easier — *but the comb has an especially simple spectrum*: the Fourier transform of a Dirac comb is **another train of Diracs** (the comb is self-dual). And convolving by a Dirac is just a **shift**, so convolving by a *train* of Diracs is just a **sum of shifted copies** — the spectrum gets **replicated** at every multiple of the sampling rate, no smearing, pure translation. So the messy-sounding "convolution" is, here, nothing but shifts — which is the whole reason sampling is so legible in the frequency domain (and the replicas are exactly what alias when they overlap; developed in [[#Sampling, downsampling, and aliasing]]).

• **Nyquist–Shannon**: to represent a signal you must sample at **more than twice** its highest frequency; above that, high frequencies **fold (alias)** down to low ones — the moiré / wagon-wheel effect [aliasing figure] [📺 embed aliasing/sampling explainer → https://www.youtube.com/watch?v=yr3ngmRuGUc, see [[Video Resources]]]

• anti-aliasing: low-pass (pre-filter) before you downsample

• **sidebar — Aliasing on purpose: dynamic moiré prints**: aliasing / moiré is usually a nuisance, but it can be **engineered**. A printed **micro-pattern** photographed by a camera produces a moiré whose appearance **encodes information** — the relative camera pose / motion — because the beat pattern is exquisitely sensitive to the sampling geometry. Cite MIT's **KinéCam** dynamic prints (https://hcie.csail.mit.edu/research/kinecam/kinecam.html). Ties to **structured / coded patterns** elsewhere in the book.

• the space-vs-frequency tradeoff (sharp in one ⇒ spread in the other)

• sidebar: a primal-domain (no-Fourier) perspective too

• perfect sinc vs practical filters. (more in resampling section)

We will discuss more in later chapter, but quickly show:

sharp input image

blurry image (say with box. Maybe just 1D ?) plus noise.

linear inversion with amplified noise.

Fourier MTF

domain operations, function from plane to itself

start with example of scaling up (upsampling). Missing values. Say downsampling will be a bit different (too many values) but the solution will be surprisingly similar.

start naive : loop over input pixels. Point out problem (gaps)

better solution: loop over output (inverse mapping: for each output pixel, inverse-transform and sample the input)

• where from vs where to: always inverse transform

challenge

derived with intuitive 1D linear interpolation

say in2D it can't be purely linear unless we triangulate

sidebar: smart triangulation where we pick the best diagonal for each square (the one joining the two most similar values)

Say we do linear horizontally first, then vertical

exercise : show it's the same if we do it in the opposite order.

• pseudocode: upsample = loop over output, inverse-map to fractional coords, bilinear-blend the four corners (two horizontal blends then one vertical)

say that the interpolation perspective is fine for linear and upsampling, but because this is about (re)sampling, Fourier has better insights. In practice, means reconstruction through convolution.

side bar: proper resampling analysis: start with sampled signal, reconstruct, prefilter, resample. The final filter is a combination of prefilter and reconstruct. Depending on whether we up or downsample one of them dominates.

• warning: much about bicubic wrong on the internet. Including in old python libraries

They all approximate a perfect sinc but with a finite support and try to find a tradeoff between loss of medium frequency and aliasing of high frequencies. Show in Fourier : perfect box, smooth approximation

Netravali Mitchell.

form of bicubic.

required symmetries, remaining two parameters

perceptual study.

point out that photoshop gives you a choice.

The key is how you scale the filter.

• 💡 **Big lesson:** **prefilter before you downsample** — when you throw samples away, first blur / area-average out the detail too fine for the new grid, or it **aliases** into bold false low-frequency moiré; naive nearest-pixel decimation is the classic bug (→ **L16**)

• (Andrew Adams' **rotation-challenge** sidebar + `fig-rotation-challenge` + the interactive `fig-rotation-resample-interactive` now sit **before** this section — they cap the reconstruction-filter-quality discussion)

• for a **shear / rotation / homography** the ideal prefilter is **spatially varying and anisotropic** — an output pixel back-projects to a stretched **ellipse** in the input, so exact filtering is costly; **mip-maps** and **EWA** (elliptical weighted average) approximate it

• the linear, **isotropic** filters so far reconstruct only inside the **Shannon/Nyquist window**; with no extra info that's all you can get

• but natural images have **regularities** (→ [[Imaging as an inverse problem]] image-manifold prior) — notably **local anisotropy**: content lies mostly **orthogonal to edges**, so the local spectrum is a **line/wedge** and the **aliases don't fill** the Fourier plane

• so we can **accept aliasing** and **steer the reconstruction** along the edge (wide along, narrow across) — equivalently, in Fourier, reconstruct only where the spectrum is and ignore the alias-free regions. Reconstruction becomes **non-linear & data-driven** (orientation must be estimated from the low-res data)

• the workhorse of **demosaicking** and older **super-resolution**: two steps — **analyze local orientation/pattern, then steer a reconstruction filter**

• **modern version**: **learn** the regularities of natural images and how to reconstruct them (→ learning chapters)

• `fig-sampling-steered-recon` [interactive] — a steered variant of `fig-sampling-2d-pipeline`: edge sampled w/o prefilter (aliases), reconstructed with an **oriented elliptical sharp-bicubic** filter (user sets orientation θ + scale along/across); the oriented passband hugs the signal's spectral line and skips the aliases → edge reconstructs; misalign → aliasing returns

• recap: resampling = reconstruct-then-resample, reconstruction = convolution with a finite-budget sinc approximation

• forward: the reduce step → Gaussian/Laplacian pyramids (next); inverse-mapping → warping & morphing; resampler quality → image metrics

• a multiresolution representation — **halfway between primal (space) and Fourier (frequency)**: localized in *both*, which is exactly why it's so practical

• multiresolution, coarse-to-fine (original papers, 1980s)

• repeatedly **blur then downsample by 2** → a tower of ever-coarser, smaller images (each ¼ the pixels): the **reduce** operator

• cheap (a geometric series → only ~4/3 the original pixel count in total); the basis for coarse-to-fine search and **scale space** [Gaussian pyramid figure]

• construction: each level = **G_k − expand(G_{k+1})** (the detail the coarser level missed) — a **band-pass** image [Laplacian pyramid figure]

• reconstruction: **exactly invertible** — add the levels back coarse → fine (an **encoder → decoder**) [encode/decode figure]

• pseudocode: Laplacian build (reduce + difference loop) and collapse (expand + add loop), each the exact inverse of the other

• frequency intuition: each level holds a **band** of detail, coarse → fine [frequency-bands figure]

• properties: **decorrelated** and **sparse** on natural images (mostly near-zero → compresses, denoises) — sparsity drives compression (JPEG 2000) and denoising (coring)

• **overcomplete** (>1 coefficient per pixel — a frame, not a tight basis); only **approximately shift-invariant**; **isotropic** (no orientation) — each limitation motivates a relative

• **wavelets**: a true tight basis (one coefficient per pixel, no redundancy) → natural for compression (JPEG 2000); but less Fourier-friendly, not shift/rotation-invariant

• sidebar: **Fourier, wavelets, and the locality dial** — Fourier (all frequency) ↔ primal grid (all space); wavelets/pyramids in between, a tunable space-vs-frequency tradeoff

• **steerable pyramids** (Simoncelli & Freeman 1995): add **orientation** (oriented sub-bands), nicely shift/rotation-invariant, even more overcomplete — texture, video magnification; (**multigrid** in passing)

• multiscale / **pyramid blending**: blend each Laplacian level under a **Gaussian-blurred mask**, recombine → seamless joins (the apple/orange) — see compositing & pano [pyramid blending figure]

• exposure fusion, HDR+ tone mapping — see HDR

• pyramid **denoising / coring**: zero out the small detail coefficients (mostly noise), keep the large ones — see Denoising [coring figure]

• local Laplacian filtering (edge-aware detail) — see edge-preserving

• coarse-to-fine (optical flow), scale (SIFT), pooling / U-Net (deep nets) — forward refs

• **full-reference**: compare against a known-good reference (denoising / compression / resampling) — most of this chapter

• **no-reference / blind**: score a single image with no reference (needs a model of natural images) — mostly set aside

• **MSE / L2** = $\frac1N\sum(I-J)^2$, simple, cheap, differentiable, but **doesn't match perception**: a tiny shift, a small gamma change, or a touch of noise can give the *same* MSE as a perceptually-awful edit (treats the image as a bag of numbers, ignores edges/texture/faces)

• sidebar: why squared (L2 vs L1) — smooth at 0, punishes outliers; L1 more robust

• **PSNR = 10·log₁₀(MAX²/MSE)** (dB) — a monotone re-expression of MSE; bigger is better (~30–40 dB decent); inherits MSE's blind spots

• sidebar: at least measure it in a **perceptual color space** (Lab / Luv, ΔE) so error ≈ perceived difference

• **SSIM** (structural similarity): compares **luminance, contrast and structure** in local windows — far better correlated with perceived quality; the *same* PSNR can have very different SSIM [SSIM-vs-MSE figure]

• **visible-difference predictors**: a full perceptual model (the **CSF + masking** from the perception chapter) that predicts *where* a difference is visible; **HDR-VDP** extends it to HDR

• forward ref: **LPIPS, DreamSim** — deep-feature distances → [[Deep learning]]

• forward ref: a differentiable metric is also a **training loss** — minimizing LPIPS/perceptual distance (not per-pixel MSE) is how learned methods get **sharp** results → [[Deep learning#Learned perceptual metrics and losses]]

• no single best metric: choose by what you're evaluating (fidelity vs perceived quality vs a training loss)

• **framing — denoising is the quintessential ill-posed inverse problem**: $\mathbf{y}=\mathbf{x}+\mathbf{n}$ with $A=I$ (point back to [[02 Fundamentals#Imaging as an inverse problem]]); the measurement constrains **nothing**, so it's impossible without extra knowledge. **Three ways** to get it (every method is one or a blend): **(1) better hardware** (more light / lower noise at the source — bigger sensor, quieter pixel, longer exposure); **(2) more measurements** (burst → make the matrix **taller**, over-determine); **(3) a prior** — **explicit** (smoothness/sparsity penalty) or **embedded** in a smart algorithm (bilateral, learned net) that assumes a prior without naming it. This chapter is mostly (3): averaging neighbours *is* a prior.

• sources: photon/Poisson, read, thermal (see noise / image formation)

• look at it: histogram of a flat grey patch, a scanline

• reminder: affine model (in linear space) is good.

• reminder: sensor saturates on both ends

• 1 → 3 → 5 images: the noise visibly drops [averaging-convergence figure]

• the statistics: independent noise averages out — variance ∝ 1/N, so **noise drops as 1/√N**; SNR, PSNR improve accordingly

• needs **alignment** first (brute force; → burst / HDR+, see multiple-exposure)

• no other measurements of the same pixel → use redundancy *inside* the image: neighbours are near-repeated measurements (same 1/√N win), but averaging smears edges

• everything below = a different answer to "which neighbours count as the same value?"

• spatial averaging: Gaussian blur — and why it **smears edges** [denoising before/after figure]

• bias–variance trade-off: wider filter cuts more noise but blurs more

• **median** filter: ignores the odd wild neighbour; good against speckle / salt-and-pepper

• sidebar: optimal *linear* filter = **Wiener** (per-frequency keep ∝ signal/noise) — proof we must leave fixed linear filters

• bilateral filter (edge-preserving) — see edge-preserving chapter — weight a neighbour by spatial *and* value closeness; per-pixel normalization, non-linear, not shift-invariant; color = distance in color space

• 💡 **Big lesson:** **edge-preserving = affinity** — use the **color / intensity difference** between two pixels as how much they "belong together"; the bilateral filter weights neighbours by that affinity.

• debug by pushing the range σ to extremes (wide → Gaussian, narrow → no smoothing); set width by the noise level

• **non-local means**: drop "nearby" — average pixels whose *surrounding patch* matches, anywhere in the image (a bilateral filter in patch space)

• **BM3D**: group similar patches, filter jointly in a transform domain — the long-time benchmark

• sidebar: wavelet / pyramid **coring** — zero the small (noisy) detail coefficients, keep the large (edge) ones — see pyramids

• sidebar: **learned denoisers** (U-Net) beat BM3D; shade into generative priors; **diffusion models** are trained denoisers — see ML

• band-pass (detail) coefficients are **sparse** on natural images → **coring**: attenuate the small coefficients (zero / soft-shrink), keep the large ones, reconstruct — flats smooth, edges survive.

• still shipped because it is **finely controllable**: a per-band, per-scale (per-orientation) threshold knob — denoise the noisy fine scales hard, leave coarse structure alone; **hard** (crisp, can ring) vs **soft** (smoother) per level. The principled version models the heavy-tailed coefficient stats — Simoncelli's **BLS-GSM** (Portilla et al. 2003).

• figures: `fig-pyramid-denoise-demo` (live per-band coring), `fig-pyramid-denoise-result`.

• eyes have coarser acuity for color than brightness → chrominance noise is uglier and safe to smooth hard

• denoise in **YUV**: gentle on luminance, aggressive (large radius) on chrominance — kills color blotches with no visible cost; ties to perception

• you need the noise level to denoise well: **calibrate** it (shoot a flat field at each ISO → a per-ISO noise model), or **estimate from the image** (the std of flat patches / a high-frequency residual)

• there's a floor: **Levin et al.** bound how much *any* denoiser can recover — once noise swamps the local structure, detail is unrecoverable; stronger **priors** (and more photons) are the only way past it

• 💡 **L4.22 — the average is safe, and the average is dull**: MMSE = the **mean** of all images consistent with the data; over genuine uncertainty that average is **blurry / washed-out**. *Two perspectives on why* — (1) **convexity / geometry**: the natural-image set is **non-convex**, so a convex average of two valid images falls **off the manifold** into the blurry gap (the manifold is messy); (2) **perception**: the squared-error ruler MMSE uses treats image space as **Euclidean**, but **human perception is not Euclidean in image space** (a global shift is $L_2$-huge yet perceptually trivial; wrong texture is $L_2$-tiny yet glaring) — so the metric is mismatched too. Escape = change the objective (perceptual/adversarial loss) or **sample** one sharp outcome (diffusion) instead of averaging.

• **side bar (ethics) — when denoising becomes beautification**: pushed too far, denoising (especially on **skin**) erases **texture** (pores, fine lines) → the **"plastic" / wax-figure** look; skin-smoothing, blemish-removal and **beautification** are the *same operation* dialled up (suppress high-freq variation in detected skin). Phones often apply it **by default**, non-consensually — silently imposing a flawless-skin standard, making the photo unfaithful to the person, and affecting **self-image** (esp. young people). Where does a denoiser stop cleaning an image and start editing a face? → ties to ethics ([[Human factors and the art of photography]])

• **quad-Bayer and Nonacell sensors** (2×2 or 3×3 same-color groups — see Sensors) require a **CFA-conversion front-end** step before the normal demosaicking pipeline:

• **full-resolution mode**: a **remosaic** maps each 2×2 (or 3×3) group back to a single-sample-per-site standard Bayer mosaic (e.g. picks the top-left sub-pixel of each group); standard demosaicking then runs as normal

• **binning / low-light mode**: each group is **summed or averaged** into one pixel, producing a quarter- (or ninth-) resolution standard Bayer image; demosaicking follows on that smaller, less noisy mosaic

• the key point: **normal Bayer demosaicking algorithms receive a standard mosaic** either way — the remosaic / binning stage is a CFA-format adapter that sits ahead of it

• → cross-ref Sensors quad-Bayer bullet ([[02 Fundamentals#Sensors photosites CCD vs CMOS]])

• one color per photosite (CCD / CMOS); the **RGGB** pattern; twice as many greens (luminance acuity) [Bayer mosaic figure]

• RAW files: the mosaic **before** demosaicking (linear)

• treat the 3 channels as sparse images, interpolate each (= upsampling); missing green = average of 4 measured neighbours

• works, but artifacts: **zippering** and **color fringing** at edges [demosaick before/after figure]

• **debugging / hand-checkable inputs**: a **constant** image must come back unchanged (nothing to interpolate); a **rectangle** on a flat field makes the **zipper / fringing** appear right at its edges — the minimal inputs that expose demosaicking errors

• a skipped pixel's 4 neighbours straddle the edge → averaging mixes the two sides → values oscillate (the zipper); root cause = **averaging across the edge**

• interpolate along edges, not across them: pick the axis whose two neighbours agree most (smaller |up−down| vs |left−right|)

• bets on a piecewise-smooth world; collapses the zipper

• 💡 **Big lesson:** **let the data pick the direction** — adapt the operation to local structure (precursor to bilateral / affinity)

• R/B are sparser (¼ sampled); harder edge direction

• even perfect per-channel interpolation gives **color fringing**: the 3 channels are sampled at different locations, so their edges don't line up (orange one side, cyan the other) — a between-channel problem

• color channels are highly correlated: **color (R−G, B−G) varies slowly**, brightness has the sharp edges

• recipe: reconstruct green first (edge-directed), form R−G / B−G at measured pixels, interpolate those *naively* (safe — smooth), add green back

• 💡 **Big lesson:** **luminance carries detail, chrominance is smooth** — green ≈ luminance (interpolate carefully), differences ≈ chrominance (interpolate crudely)

• sidebar: difference vs ratio (constant-hue); divide-by-near-zero risk

• still not perfect: fine color texture → **maze / labyrinth** artifacts

• the **OLPF / anti-aliasing filter**: a birefringent layer that slightly blurs before sampling (low-pass before sampling, from the aliasing chapter) — trades sharpness for fewer color artifacts; weakened / dropped as demosaicking improved (computation displacing optics)

• ⚠️ **increasingly omitted** on modern cameras: **higher pixel counts** make aliasing / moiré less visible, and buyers want **maximum sharpness** — a deliberate **sharpness-vs-moiré trade** (drop the filter, accept occasional moiré)

• RAW is noisy; demosaicking and **denoising** are entangled → do them **jointly** (better than in sequence)

• learned joint denoise+demosaick networks (Gharbi et al. 2016; Adobe Enhance Details) — forward ref to ML

• temporal multiplexing (Prokudin-Gorskii, color wheel) — static scenes only

• spatial multiplexing (the mosaic itself, the retina's cone mosaic), 3-chip prism (3-CCD)

• depth multiplexing (film tripack, Foveon)

• ties back to the Color technology

• one of the *first* ISP stages (after black-level / defective-pixel), on linear RAW, **before** white balance / color matrix / tone / sharpen / gamma — order matters

• **the AE problem**: pick aperture/shutter/ISO so the scene lands at the **middle-grey** (~18%) target; the catch is *which* pixels are representative

• **average / center-weighted** metering — average the frame (fooled by bright skies → silhouettes, snow → grey); center-weighting assumes a centered subject

• **matrix / evaluative / pattern** metering — split into **zones**, combine brightness + contrast + focus point (+ color + lens distance = Nikon's **3D Color Matrix**); historically a lookup against a database of reference scenes; the default today

• **spot** metering — meter one small zone; the photographer chooses the anchored tone (Adams's Zone System)

• **learned** metering — a small network (often + semantic segmentation: sky/face/backlight) predicts the photographer's exposure; phones choose exposure jointly with burst/HDR (deliberately under-expose, recover shadows by stacking → Multiple exposure), leaning on the affine+truncated noise model

• 💡 same arc as AWB below: one hand-coded statistic → multi-zone pattern model → learned estimator, all guessing intent from pixels

• spectral perspective

• Von Kries

• **white balance is a per-channel multiply — but only correct in *linear light***: it scales each channel by a gain (Von Kries), and a scaling **commutes with a *linear* map but not a non-linear one**. Apply the same gains to **gamma- / tone-curve-encoded (non-linear)** values and you get the *wrong* answer — the channel gains then **shift hue and change apparent saturation**, not just neutralize the cast. So white balance belongs **early, in the linear pipeline**, alongside exposure (also a multiply-in-linear-light). [fig-wb-linear-vs-nonlinear — the same white-balance gains applied in linear light vs. on gamma-encoded values: the hue/saturation shift] (→ [[#Linear vs gamma vs. log encoding]])

• **white balance from color temperature and tint (the raw-converter slider model)** — the two sliders just pick a **target illuminant white point**, and the diagonal gains follow:

1. **Temperature** $T$ (Kelvin) selects a point on the **Planckian (blackbody) locus** — its chromaticity $(x_c(T), y_c(T))$ from a standard closed-form fit (Kim et al. 2002): warm/low-K → orange, cool/high-K → blue.

2. **Tint** nudges that point **perpendicular to the locus**, along the **green↔magenta** axis (a signed $\Delta uv$ in CIE-1960 $uv$) — correcting casts a pure blackbody can't, e.g. fluorescent green.

3. Convert the white point to tristimulus: $X = x/y,\; Y = 1,\; Z = (1-x-y)/y$, then to the working/sensor space $\mathbf{W} = M_{XYZ\to RGB}\,(X,1,Z)$.

4. Set the **Von Kries gains** $k_c = G_w / W_c$ (normalize to green) so a surface under that illuminant reads **neutral**.

So "$T=5500$ K, tint $0$" and "$(k_R,k_G,k_B)$" are **two faces of the same diagonal correction** — the sliders are a human-friendly parameterization of the per-channel gains. (M lives in [[#Reproducing color]]; *where* the multiply must happen is [[#Linear vs gamma vs. log encoding]].)

• old style: grey assumption, bright pixels

• 3D matrix regression style

• modern ML solutions

• **skin tones** — skin is a **memory colour** cameras are tuned around (face detection feeds the meter + AWB so the *person* lands right); but that tuning has a **history of bias** (the **Shirley card** calibrated film/AE/AWB to light skin, rendering darker skin poorly). Obligation: design & test AE/WB across the full range of skin tones → ethics in [[Ethics of computational photography]]

• multiple / mixed illuminants: one global per-channel gain can't fix a scene lit by two light colors (forward ref: multi-light WB, Hsu et al.)

• metamerism failure: WB can't undo a bad illuminant spectrum (narrow-band fluorescent / cheap LED)

• Color Rendering Index (CRI): how faithfully a source renders colors vs a reference illuminant

uncompressed/raw → lossless (PNG, TIFF) → lossy (JPEG)

lossless = exploit statistical redundancy; lossy = also discard perceptually unimportant info (ties to perception chapter)

• stuff like name, image size

• capture settings (ISO, shutter, aperture, focal length), GPS

• orientation flag (the infamous "why is my image sideways" bug)

• ICC color profile (ties to color management)

• typical-fields **catalog → appendix [[Common EXIF fields]]**; the practical caveats (coverage incomplete & inconsistent; ISO / shutter approximate & not radiometric) are introduced in [[#Image representation]]

-Fun with EXIF: my kids growing up, photos from X years ago

• we use PNG because it's easy to read

• mention alphas in passing to warn people (because they need png with no alpha)

Central idea : leverage human perception (cite early chapter) to adjust the number of bits to what matters. Remind people of CSF and different color sensitivity

• **framing**: JPEG is a perfect illustration that **perceptual criteria can be made quantitative and objective** — "what the eye won't miss" becomes a concrete **quantization table** and bit allocation rather than hand-waving; perception turned into numbers you can compute, tune (the quality knob), and measure

• the **quality knob** = a multiplier on the quantization table; show a **quality sweep** of the same photo (q ≈ 85 → 5) with the resulting file size under each — degradation (softening → blocking → color bleed) grows as the file shrinks [JPEG quality-levels figure]

#### Stage 1: opponent color

Opponent color space

#### Stage 2: chroma subsampling

chroma subsampling (4:2:0 / 4:2:2): coarser chroma than luma because color CSF is lower

#### Stage 3: the 8×8 DCT

DCT

#### Stage 4: quantization — the lossy step

quantization table

• aside — **8-bit pixels carry every DCT coefficient to ≈8 bits, frequency-independent** [`fig-dct-8bit-roundtrip`]: render one basis coefficient → quantize pixels to 8 bits → transform back; recovered ≈ intended to ~8 bits for low/mid/high frequency alike, because the DCT is an orthonormal rotation (white pixel-rounding noise spreads evenly over all 64 coefficients). So JPEG's coarse high-frequency quantization is a **perceptual** choice (CSF) + energy statistics, *not* a pixel-storage limit.

#### Stage 5: entropy coding

classic entropy coding, with smart block order

#### Limitations and artifacts

limitation: 8-bit only (motivates other formats); artifacts: blocking, ringing

• **generation loss is largely overblown:** re-saving at a fixed quality settles into a **fixed point** (coefficient quantization ≈ idempotent) and stops degrading — same resilience as the repeated-quantization experiment of 3.1 (in theory it compounds, in practice it's stable). [`fig-jpeg-generation-loss`]

• **combined worst case** — the §3.1 iterated-exposure loop *with* a JPEG round-trip at every step: 8-bit banding and DCT blocking/ringing compound together, still tending to a fixed point. [`fig-exposure-jpeg-roundtrip`]

• **exception — deblock-on-load drift:** software that prettifies low-quality JPEGs on load (Photoshop's deblocking) breaks the fixed point → the load→save loop can **drift** [@hackerfactor-himym] (▶ youtube Fk6kV5N1rzs); a filter-free codec does *not* drift (▶ youtube gJJachY651c).

#### Removing the artifacts: JPEG deblocking

• post-hoc **deblocking / artifact reduction**: POCS onto quantization intervals (Zakhor 1992 [@zakhor-1992]); shape-adaptive DCT (SA-DCT, Foi et al. 2007 [@foi-etal-2007]); in-loop deblocking in video codecs (H.264, List et al. 2003 [@list-etal-2003]); learned CNNs (AR-CNN, Dong et al. 2015 [@dong-etal-2015]). All trade fidelity for appearance → decode is no longer a faithful inverse of encode (the drift above).

usually before demosaicking

linear

proprietary vs DNG

• **"RAW" is not always purely raw**: many cameras pre-cook it to some degree — black-level subtraction, defective/hot-pixel correction, sometimes lens-shading/vignetting correction or even mild denoising; some "RAW" is lossy-compressed or non-linearly encoded. So RAW is a *spectrum* of how cooked it is, not literally the untouched sensor readout.

• tools/libraries to read RAW: **dcraw** (the classic), **LibRaw** (C++, dcraw's successor) and its Python wrapper **rawpy**; open-source developers **darktable** / **RawTherapee**; **Adobe DNG Converter / DNG SDK**; **exiftool** for metadata; commercial **Lightroom / Capture One**.

quickly in passing

EXR (OpenEXR), Radiance .hdr, 16-bit TIFF — needed to store HDR output (see HDR chapter)

HEIC/HEIF (Apple default now, 10-bit / HDR-capable), WebP, AVIF, JPEG XL

• **the idea**: replace the hand-designed transform + quantizer + entropy coder of JPEG with a **trained autoencoder** — an **analysis** network maps the image to a latent, the latent is quantized and entropy-coded under a **learned prior**, a **synthesis** network reconstructs. The whole pipeline is optimized **end-to-end** for a **rate–distortion** objective (bits vs. a distortion/perceptual loss), so the codec *learns* what to throw away.

• **why it matters**: at low bitrates learned codecs beat JPEG/even AVIF on perceptual metrics, and they can optimize for **perceptual / GAN** losses rather than MSE (sharper, more natural reconstructions at the same rate). They also generalize: **video**, **depth**, and **scientific** data, and the latents double as features for downstream tasks.

• **the catches**: heavy compute (a neural net to *decode*, not just encode), **non-determinism / version-coupling** across encoder–decoder, and no universal standard yet — though **JPEG AI** (ISO/IEC) is the first standardized learned image codec. Ties to [[Deep learning]] (autoencoders, entropy models) and to generative priors in [[Single-image computational photography]] (super-resolution, inpainting are the same rate–distortion-with-a-prior idea pushed to extreme compression).

• photoshop, layers

• mention alpha again in passing.

• the **image signal processor** runs a fixed **pipeline** from RAW to JPEG; the **order matters** — mostly **linear light** until the end [ISP block-diagram figure]

• **3A** (auto-exposure, auto-white-balance, auto-focus — AF in Optics); **black-level** subtraction; **hot / defective-pixel** correction

• **demosaick** (→ ch) · **denoise** (→ ch) · optional **lens corrections** (vignetting, CA, distortion)

• **white balance** + **color matrix** (sensor → standard RGB) · **tone mapping / curves** (incl. local) · **sharpening**

• **gamma encoding** · **beautification curves** · **JPEG** compression (→ ch)

• advanced: **aberration correction**, computational stages (HDR+, night mode) — later parts

• the throughline (Big lesson): do the physics (WB, demosaick, denoise) in **linear**, **encode (gamma) near the end**

• e.g. **Adobe Lightroom**: keep the **RAW** untouched and store edits as **parameters** (a recipe — exposure, contrast, curves, masks), recomputed on the fly

• benefits: **reversible, re-editable, resolution-independent**; needs a **fast (GPU) pipeline** to recompute interactively

• the parametric pipeline ≈ a programmable ISP exposed to the user