2.16 Limitations of the medium🔗⧉

We have spent FOUNDATIONS building a picture up from physics: light, a lens, a sensor, noise, color, the camera. Now step back and ask the opposite question: what can a picture, as a medium, never do? A photograph or a painting is a flat, finite, single-viewpoint, static, low-contrast, limited-gamut surface. The world the eye moves through is none of those things. Every one of those mismatches is a limitation of the medium, and this chapter's organizing idea, taken from Durand's depiction course, is that the entire craft of picture-making is the art of responding to them. Every response falls into one of three categories. You can compensate: push a cue that does survive the medium harder, so it stands in for one that doesn't. You can accentuate: lean into what the medium is good at. Or you can deliberately set cues in conflict, for effect. Computational photography is, to a surprising degree, the same three moves with an algorithm doing the work.

There is a deeper way to say why this matters, and it comes from the NPAR paper (Durand 2002): depiction is "the inverse of an inverse problem." Vision is already an inverse problem — the brain infers the three-dimensional world from the flat pattern of light on the retina. A picture has to manufacture a new pattern of light, within the medium's limits, that drives the same inference. That is why "the direct recording of the optical flow — i.e., photography — might not result in the most realistic image." Sometimes the faithful thing to do is not to record what was there but to reinforce an occluding contour, or add a glow around a lamp, so that the viewer's visual system reconstructs the right scene. These limitations are constraints that shape depiction.

The rest of the chapter walks the limitations one at a time — flatness, time, the frame, dynamic range — and ends on resolution, the one limitation modern capture has essentially beaten, precisely to throw the others into relief (Figure 2.16.1).

2.16.1 Compensation, accentuation, conflict🔗⧉

The medium is specific, and its specificity is a short list: a picture is flat, has a frame, shows one viewpoint, is usually static, and is limited in contrast and gamut. Hold that list in mind; the whole chapter hangs on it. For each item, the maker compensates, accentuates, or sets up a conflict. A landscape painter, denied stereo vision, over-emphasizes linear perspective and aerial haze (compensation). A photographer, unable to emit real sunlight, adds a flare around the sun (compensation by a learned cue). Magritte, given a flat surface on which size is a free variable, paints an apple that fills a room (conflict, on purpose). The same surface that loses the third dimension licenses the impossible — which is exactly why the medium is interesting.

Stated as a clean menu, faced with any one of these limitations you have three responses:

1. Accentuate and leverage it — lean into what the medium does well, and even turn the limitation into an expressive tool (the flat surface that licenses Magritte's room-filling apple; the frame as composition; the frozen instant as a "decisive moment").
2. Compensate — push a cue that does survive the medium so it stands in for one that doesn't (perspective and haze for the missing stereo; a painted flare for the sun you cannot emit; a tone curve that fakes the look of high contrast).
3. Break it — refuse the limitation and build technology that actually removes it. This is increasingly the computational-photography move, and it organizes much of this book: HDR capture and HDR displays attack the contrast limit (FUNDAMENTALS — dynamic range; HDR merging); panoramas and VR attack the frame and the single viewpoint (panorama stitching; immersive displays); light-field / integral photography and immersive, glasses-free 3-D displays attack flatness itself. Where depiction works within the limit, computational imaging tries to abolish it — and the new part Integral and immersive imaging (right after 3-D and depth) is devoted to that third response on the flatness/viewpoint axis: stereo and VR displays, lenticular and light-field screens, and holography, the technologies that try to give a picture back the depth the medium took away.

2.16.2 The picture is flat: depth and its cues🔗⧉

The first major limitation is flatness: a picture collapses the third dimension onto a plane. We recover depth from a surprisingly long list of cues, and the question that organizes everything is which cues survive projection onto a flat surface and which do not. (Helmholtz called the underlying process unconscious inference — the brain infers depth from these cues without our noticing; Durand's course assigns his Relation of Optics to Painting.) Figure 2.16.2 lays the whole catalog out.

The cues a flat picture cannot reproduce are the ones that need two eyes, a focusing eye, or a moving viewer:

• Binocular disparity (stereopsis) — the two eyes see slightly different images, and the brain decodes the disparity into depth. Julesz's random-dot stereogram proves the channel is real and purely geometric: two fields of random dots, identical but for a shifted central patch, fuse into a floating square with no monocular form to give it away (Julesz 1971; Figure 2.16.12). A flat print sends both eyes the same image, so this cue is simply dead.
• Convergence — the eyes rotate inward to fixate a near point; the angle is a distance signal, but only out to a few meters.
• Accommodation — the eye's lens changes shape to focus at a distance; a weak cue, near range only. Its visual shadow, defocus blur, does survive in a photograph (below), but the muscular cue itself does not.
• Motion parallax — when the viewer moves, near things sweep across the field faster than far things. A still image cannot deliver it; a video can (→ VIDEO) (Figure 2.16.13).

So why does a flat picture read as three-dimensional at all? The decisive demonstration is the pseudoscope, a device that swaps the two eyes' images and thereby reverses stereo. Looking at a real scene with strong pictorial cues, the brain refuses to invert — it overrides the contradicted disparity and trusts the pictorial cues (occlusion, shading); only when those cues are weak does the reversed stereo win. The lesson is blunt and liberating: pictorial cues are stronger than stereo. That is the empirical license under which a painter or photographer can build convincing depth on a flat surface, and it is why the rest of this section matters.

The pictorial cues a flat picture can carry — the painter's toolkit, and what good depiction over-emphasizes to compensate for the missing cues above — are:

• Occlusion (overlap) — the strongest, most robust depth cue, and a purely ordinal one: it tells you front-from-behind, not how far. (The overlapping animals of Lascaux already use it.) This is the cue NPAR singles out as the one to reinforce when stereo is unavailable (Figure 2.16.4).
• Relative size — equal objects shrink with distance; a receding row of identical figures reads as a line into depth (Figure 2.16.5).
• Familiar size — we know real-world sizes, so apparent size sets distance — and because the cue is so strong, it is the easiest to subvert (Magritte's room-filling apple; the forced-perspective tourist "holding" the leaning tower) (Figure 2.16.5).
• Height in the visual field — an object whose base sits higher in the frame (up to the horizon) reads as farther; medieval painting, Chinese scrolls, and Bruegel's winter scenes stack figures up the field to push them back (Figure 2.16.6).
• Linear perspective — parallel lines converge to vanishing points (Raphael's School of Athens is the textbook one-point case). This is the projective geometry of pinhole image formation, used as a depth cue (Figure 2.16.7).
• Shading, and the light-from-above prior — luminance gradients model form, and the brain reads them under a hard-wired assumption that light comes from above. Invert a photograph of shaded bumps and they pop into dimples; turn a crater photograph upside-down and the craters become domes; a concave plaster face mask still looks convex (the hollow-face illusion). A flat picture exploits this prior, and the reversal makes it visible (Figure 2.16.3).
• Cast shadow — a shadow anchors an object to the ground plane and signals its height. In the Kersten–Mamassian ball-in-a-box demonstration, a single ball appears to rest on the floor or to float in mid-air depending on nothing but the position of its shadow (Kersten et al. 1997; Figure 2.16.8).
• Texture gradient — a statistically uniform texture (pebbles, furrows, grass) compresses with distance, giving a surface its slant and recession (Constable's fields) (Figure 2.16.9).
• Aerial (atmospheric) perspective — scattering in the air makes distant things lose contrast and saturation and turn bluish (Leonardo's hazy blue backgrounds). Its subtler role, per NPAR, is grouping: tinting everything at one distance with a common property binds it together (Figure 2.16.10).
• Defocus blur (depth from defocus) — a photograph can bake in depth of field, throwing the background into a soft wash while the subject stays sharp, and we read the blur as depth (lens image formation). But notice what it costs: the blur is the photographer's choice, frozen into the print, and it denies the viewer the ability to refocus — a manipulation we return to in Photographs are usually not passive objective recordings, and a dimension that light-field imaging later hands back (Figure 2.16.11).

Finally, because the pictorial cues are independently controllable, an artist can set them in conflict on purpose — the hollow-face reversal, Escher's figures that are locally consistent but globally impossible, Magritte's scale games. Flatness is not only a loss of information; it is a license to depict what cannot exist.

2.16.3 The picture is static: time and motion🔗⧉

A single photograph freezes one instant; duration, change, and motion are gone. And yet a still image conveys movement through a rich vocabulary, which splits naturally into what you can do within one frame and what you can do by combining several.

Within a single snapshot, the devices are:

• Telling pose — deviation from rest. A body caught away from its resting position reads as in motion, because we sense it "wants" to fall back: the skater at the top of a leap, Géricault's horses in the (anatomically wrong but pictorially convincing) "flying gallop," the Discobolus coiled to throw versus a frontal, static Egyptian figure. The reading is partly innate — developmental studies find that this deviation-from-rest cue is read strongly even by young children and, if anything, fades with age, whereas the drawn motion-line convention (below) is the one children learn to read as they grow; either way, the more dramatic the pose, the stronger the effect.
• Motion blur — the streak a moving subject leaves during a finite exposure. A painter anticipated the camera here: Velázquez blurred the spokes of a spinning wheel in Las Hilanderas centuries before a shutter made the effect automatic.
• Trails — the long-exposure light-trail: car tail-lights drawn into ribbons, or Andreas Feininger's helicopter rotor-tips arcing into glowing circles.
• Action lines and path — the comic-strip convention of speed lines and a drawn trajectory (Keith Haring made the radiating "movement lines" into fine art), and the dance-notation diagram that draws the path of the feet.
• A dynamic viewpoint — a low, tilted, wide angle that injects energy into an otherwise frozen frame.

Combining several instants takes two forms, and the distinction is worth keeping. The first is capture: Muybridge's sequential gallop plates — which famously settled whether a galloping horse is ever entirely airborne (Muybridge 1887) — Marey's chronophotographs laying multiple phases on one plate, and Edgerton's stroboscopic high-speed flash (the bullet splitting an apple, the milk-drop coronet). These recover the time axis the single frame threw away, and they are the direct ancestors of multiple-exposure imaging and high-speed video. The second is depiction: the Futurists — Balla's Dynamism of a Dog on a Leash, Duchamp's Nude Descending a Staircase — paint many phases of motion at once on a single static canvas. Capture samples time; depiction represents it (Figure 2.16.14).

The computational sequel to this whole limitation is the video part of the book: recover the lost dimension by sampling time densely, and then exploit it — frame rate, motion blur, frame interpolation, even amplifying changes too small to see (video magnification) → VIDEO.

2.16.4 One viewpoint, and a finite frame🔗⧉

Two limitations travel together. A picture is taken from one fixed station point — unlike the moving, two-eyed observer — and it has a hard boundary, a limited field of view, so everything outside the rectangle is simply cut away.

A single viewpoint can be ambiguous: from one viewpoint a shape may read correctly, and from another it may mislead (every forced-perspective gag lives here). Picture makers have spent centuries smuggling extra viewpoints into the frame — a mirror that shows a second view (Velázquez's Rokeby Venus), the engineer's multiple orthographic views (plan, elevation, section), exploded and reverse/divergent perspective (Byzantine icons, Hockney's photo-joiners), and finally Cubism, which fractures a single face into many simultaneous views. Behind all of it is the tension Durand frames as "what I see versus what I know" — Turner's "I paint what I see" against Picasso's "I paint what I know."

The frame itself is the other half. A picture is a window — Alberti's finestra aperta, the rectangle Dürer's draughtsmen drew through with a gridded screen — a finite opening with a sharp edge the eye never experiences in life. The maker chooses the cut: composition, cropping, and the panorama, which stitches a wide field to escape the frame (and which needs curvilinear perspective, because straight-line perspective explodes past about 90°). Put numbers on the gap and it is stark: human vision spans roughly 180–200° horizontally, but only about 2° of it is sharp foveal vision, swept across the scene by saccades (human vision); a normal lens and print subtend maybe 40–50°. The picture is a narrow, hard-edged porthole onto a panoramic, edgeless world. And field of view is bound up with focal length and perspective (lens image formation): a wide lens crams in a huge field but distorts a near face; a long lens narrows the field and flattens depth (Figure 2.16.15).

2.16.5 The contrast is limited: dynamic range and gamut🔗⧉

The real world spans about 10⁻⁶ to 10⁶ cd/m² — luminance, loosely the brightness of a surface, measured in candela per square meter — roughly twelve orders of magnitude across conditions, from starlight to a sunlit snowfield — while a print delivers at best about 1:500 contrast (often only 1:50) and a display lives between about 1 and 100 cd/m². The scene's range exceeds the medium's range (Figure 2.16.16). This is the same dynamic-range story told quantitatively in noise and dynamic range, now from the output side.

There are really two distinct problems hiding in "contrast is limited." The first is intensity mismatch: a sunny scene is viewed in a dim room, so the picture's actual luminances live in a different, lower band of the scale than the scene's ever did. The second is insufficient contrast: even after you place the picture in the right band, a high-dynamic-range scene has to be compressed into the narrow ratio the medium allows.

Why does the eye so badly out-perform the medium? Because the visual system judges local contrast and reflectance, not absolute luminance — local adaptation plus center-surround edge response, the picture Land formalized as Retinex (Land & McCann 1971). We read deep shadow and bright highlight in the same scene that no single exposure can hold, because we are computing ratios across edges, not measuring absolute brightness. The medium has to fake that, and the history of picture-making is full of the attempt. Andrea Pozzo's painted sky on the ceiling of Sant'Ignazio simply "is not bright enough" — no pigment reaches real sky luminance — and Brunelleschi, in his founding perspective demonstration, left the sky as polished silver to reflect the actual sky, a frank admission that paint could not get there.

The compensations are, in embryo, the modern toolbox:

• Exposure and metering place the scene's range inside the medium — center-weighted, spot, and incident metering, up to multi-segment "matrix" metering (the exposure chapter).
• The Zone System (Ansel Adams) pre-visualizes the scene's tones and deliberately places them within the printable range — the photographer's manual dynamic-range compression, roughly a 1:10,000 scene squeezed to a 1:300 print (Adams, The Negative).
• Dodging and burning lighten and darken locally during printing — W. Eugene Smith spent days dodging a single Schweitzer print — and edge burning darkens the borders to hold the eye. This is the manual ancestor of every local tone operator.
• Tone reproduction uses film and print's gentle toe–linear–shoulder curve to roll off shadows and highlights into the available range, together with gamma (ISP).
• Perceptual and time-dependent tone mapping aims to reproduce a faithful impression rather than physical values — matching the smallest perceptible difference, simulating glare and slow dark adaptation (Ferwerda et al. 1996, Ferwerda et al. 1996). The whole HDR / tone-mapping chapter is this compensation, automated.
• Flare, halo, and glow added around a bright source suggest a brightness the medium cannot physically emit — a learned "this is very bright" cue, and one of the pictorial techniques NPAR lists for compensating (Durand 2002).

The gamut limitation is the color sibling of limited contrast: the medium works from a restricted set of primaries or pigments that cannot reach every real color, handled by gamut mapping (color technology). Same shape of problem — a large real space projected into a small reproducible one — and the same need to compress gracefully rather than clip.

2.16.6 Resolution: the limitation we are overcoming🔗⧉

One more axis, and the optimistic one. The eye's foveal acuity is about 1 arc-minute — on the order of 30 cycles per degree (thirty light/dark line-pairs packed into one degree of visual angle; the finer ~60-cycle figure is the cone-spacing sampling limit) — and a picture is uniformly sampled at a fixed resolution (where the eye is sharp only in that tiny fovea and blurry everywhere else, steered by saccades). Of all the medium's limitations, spatial resolution is the one capture and display have essentially caught up to: high-megapixel sensors and "Retina"-class prints and screens now reach or exceed foveal acuity at a normal viewing distance, so a still photograph can already be indistinguishable from reality in sharpness.

That is the chapter's final point. Resolution is the success story, and naming it as solved throws the persistent limitations into relief — dynamic range, the lost third dimension, time, the finite frame, gamut. The rest of this book is, in large part, the ongoing campaign against those. And the companion chapter that follows turns the lens around once more: having seen what the medium cannot do, we ask what the photographer inevitably does — because there is no such thing as a passive, objective recording.