1.1 From Digital to Computational Photography🔗⧉

The invention of digital photography swapped a roll of film for a sensor. But the sensor brought something far more consequential than instant gratification. Once the image is a grid of numbers, you can put arbitrary computation between the photons in the scene and the picture you finally look at. The camera stops being a copying machine and becomes a small computer with a lens and a sensor attached. The possibilities this opens are what this book is about.

Computation lets us do great things. We can photograph a scene whose range of brightness far exceeds what any single exposure can hold, like a sunlit window seen from a dark room, and recover detail everywhere. We can computationally remove blur from a shaky handheld shot. We can stitch a sweep of ordinary frames into a single wide panorama no lens could capture in one shot. And if we capture not just where light lands but the full bundle of rays flowing through the lens, we can refocus a photograph after it was taken, or pull a 3D model of the scene out of it.

Even the most ordinary snapshot already depends on computation. As we'll see shortly, a typical sensor measures only one color channel at each pixel (red, or green, or blue), and the other two are reconstructed by an algorithm before you ever see a full-color image. Computation isn't an afterthought for fun special effects. It has become a fundamental aspect of image formation.

This is why a phone takes far better pictures than its tiny lens and sensor have any right to produce. Squeezed into a few millimeters, the optics are mediocre by any classical standard; the image quality comes from computation: aligning and merging many frames, learned denoising, and careful tone reproduction do the work that a big lens used to do.

And the story is still accelerating. Generative AI has learned to model the space of all plausible images, and can now synthesize, complete, and transform pictures with a fluency that would have seemed like science fiction a decade ago. This is conceptually unsettling: a machine that has, in effect, internalized every photograph that could exist, much like the library in Borges's story that contains every possible book (Borges 1941; on the parallel between that library and modern generative models, see Bottou & Schölkopf 2023).

1.1.1 Computation is the new optics🔗⧉

For almost the entire history of photography, the principle of image formation barely changed. A lens gathers rays from a scene and focuses them onto a flat surface, first film and later a sensor, which records them directly. The final photograph is a copy of the optical image the lens forms. Under this arrangement there is really only one way to make a better picture: build better optics. Sharper lenses, finer film, larger formats. Quality was an optics problem, and for a century and a half that is exactly how the field improved.

Computational photography breaks this chain. By inserting computation between the optical image and the final picture, we are no longer bound to copy what the lens produced. We can push past physical limits the optics alone couldn't beat, edit flexibly after the shutter has closed, record information an ordinary photograph throws away, such as depth, scene structure, and the direction light was traveling, and construct visual experiences that have no single-exposure counterpart at all.

The deepest consequence goes further. Once the final image need not resemble the optical image, the optical image is freed to be whatever is most useful to the computation, not what looks best to the eye. The raw measurement can be a coded, scrambled, or otherwise strange thing, as long as an algorithm can turn it into the picture we want. Computation is not merely post-processing tacked onto a normally formed image: the optics themselves get redesigned around it. The lens and the algorithm are co-designed, and we optimize the whole pipeline end to end rather than each piece in isolation.

Consider a counterintuitive way to reduce defocus blur and extend depth of field. Wavefront coding Dowski & Cathey 1995 and the lattice-focal lens Levin et al. 2009 (4D frequency / lattice-focal) modify an optical design by placing new elements that yield an optical image where all depths of the scene are blurry. But this blur is designed in a special way, so that it can be removed well through computation, unlike traditional defocus blur. As a result, the end-to-end joint imaging system, optics plus computation, can achieve performance that would be impossible through optics alone, and the special optics can be as simple as a tiled filter that screws onto an ordinary lens (Figure 1.1.3). This is largely because computation has two advantages over optics: the physics of optics limits us to positive combinations of light, and linear ones at that, whereas computation is free to subtract and to combine nonlinearly. We return to this kind of optics–algorithm co-design in Computational optics and coded imaging.

That is what we mean by the slogan "computation is the new optics." Optics were how we extended our ability to image the world for a hundred and fifty years; digital processing now vastly expands how we form and enhance images, and it does so with a flexibility that grinding glass never had.

1.1.2 The goals of computational photography🔗⧉

It helps to keep in view why we reach for computation in the first place. The motivations recur throughout the book, and a given technique usually serves more than one of them at once:

• Prettier pictures: straightforwardly better-looking results: cleaner, sharper, better-toned, more pleasing color.
• Alleviate physical limitations: beat the constraints the hardware imposes: too little light, too little dynamic range, a lens too small or too shaky.
• Record more information: capture what an ordinary photograph discards: depth, scene structure, separate layers of the scene.
• Facilitate human-driven post-processing: give the photographer more control after capture rather than locking in every decision at the moment of the shot.
• New visual experiences: make images that simply have no single-shot equivalent, like a refocusable photograph or a navigable panorama.
• Reveal the invisible: surface what the eye cannot see on its own, from tiny motions and color changes to information outside the visible band.

One caveat before we go on, because this whole book is largely about beating the medium's limits and it would be easy to mistake that for the only stance worth taking. A limitation is not always an obstacle to overcome; sometimes it is exactly what makes the picture, and the photographer's choice is to accept it rather than fight it (Figure 1.1.4). A scene of enormous dynamic range need not be tone-mapped flat: let the shadows crush to black and the few lit shapes that survive can read as a stark graphic that a faithful, full-range rendering would only dilute. And the camera's most basic restriction, that it records a single thin slice of time, is also a gift: it lets us hold still, and contemplate at leisure, an instant that itself lasted a fraction of a second and that the eye could never have studied as it happened. The computational techniques in this book are useful, but knowing when not to deploy them, when the limitation is the art, is part of the craft.

These goals are not all of one kind, and knowing which kind you are facing tells you what counts as a good answer, and what tools even apply. It helps to picture a spectrum running from objective to subjective, with a surprisingly important middle.

At the objective end live physics and measurement: radiometry, optics, sensor noise, geometry, reconstruction error. Here there is a right answer. A deblurred image is measurably closer to or further from the ground truth. A signal-to-noise ratio is a number. A calibration is correct or it isn't. You can put error bars on everything.

At the subjective end live taste and intent: what makes a photograph good, how to compose it, which color grade is pleasing, what artistic look to aim for. There is no single correct answer, and pretending otherwise just hides the judgment being made.

The interesting part is the middle ground, perceptual vision, which concerns how an image actually looks to a human eye. You might expect this to be hopelessly subjective, but it is not. Human vision is remarkably well modeled mathematically, along several measurable dimensions: the contrast sensitivity function, color appearance, lightness and brightness, visual masking, perceived sharpness, and full image-quality metrics such as SSIM and visible-difference predictors. "How it looks" can be measured and even predicted, though it is not pure physics. That is precisely what lets us engineer for the eye: tone-mapping to fit the eye's response, throwing away color detail the eye won't miss, choosing where to spend bits or where a little smoothing will go unnoticed.

This objective → perceptual → subjective spread is a lens we'll return to throughout the book. Before you reach for a tool, it pays to ask which kind of problem you are actually solving: a measurement to get right, a perceptual effect to model, or a matter of taste to respect.

1.1.3 What makes a technique useful🔗⧉

Beyond what problem a technique solves, there is the question of what makes it good to use: the engineering qualities that decide whether a method graduates from a paper to a product. Three matter most. A useful technique is fast, cheap enough to run at the scale and latency its use demands, ideally interactively or on the device. It is predictable, with graceful degradation: when its assumptions are violated it should fail gently and visibly, a little softer, a little noisier, rather than catastrophically, throwing bizarre artifacts that are worse than having done nothing. And it is controllable and tunable: it exposes a few meaningful knobs a photographer or engineer can actually turn, rather than being a black box with one fixed behavior. Methods that score well on all three tend to win, even against more accurate but brittle or opaque rivals.

But a warning about the first of those, because it is the one people over-weight: solve the problem first, then worry about speed. A technique that produces the right result slowly is far more valuable than a fast one that produces the wrong result. Speed can almost always be bought later, with better algorithms, new hardware, or approximations, whereas a method that is fundamentally wrong cannot be sped up into correctness. The history of this very field makes the point: bilateral filtering and the local Laplacian filters were both, when they first appeared, too slow to be practical. Both became everyday tools once the idea was proven and the fast approximations followed (Bilateral filtering, Local Laplacian filters; the speedups themselves are collected in Algorithmic speedups). Premature optimization is as much a trap in imaging as anywhere else in engineering: get it right, then get it fast.

1.1.4 A field in transition: from hand designed algorithms to learning to generative AI🔗⧉

Like much of computing, computational photography is undergoing a revolution. The transition runs through three stages. First, traditional hand-designed techniques, where an engineer works out the algorithm. Then supervised, deep machine-learning techniques, trained on pairs of inputs and desired outputs. And finally generative approaches, such as denoising diffusion, that can generate an image ex nihilo, out of nothing, sampling from the space of all possible images.

On the first end of this spectrum the measurements from the camera rule: we seek to tweak them to make the picture better, but we never stray too far, and we always refer to an objective reality. On the other end, generative techniques are trained as priors over plausible images; they may try to steer their output toward the measurement, but the measurement has become a second-class citizen.

This changes a long trend in image-making. From the Renaissance, to the invention of photography, to the expectation that grew from the middle of the twentieth century with photojournalism that a photograph would be the snapshot of an actual instant in time, we converged on an ideal of objective measurement, even if, as we will argue, we never fully reached it. Generative AI and text-to-image models are now revolutionizing how we create images, and time will tell how that will reshape our relationship with photographs.

This transition is a local instance of a pattern that has played out across all of AI, and it is worth naming because it shapes how to read the rest of the book. Rich Sutton called it the bitter lesson (2019): over the long run, the methods that win are the general ones that scale with computation and data, namely search and learning, not the ones that hard-code human insight about the problem. Hand-engineered cleverness gives the early lead and feels intellectually satisfying, but it is repeatedly overtaken, once compute is cheap enough, by a less clever method that simply learns from more data. The canonical cautionary tale is from natural-language processing: for decades, parsing a sentence meant hand-building elaborate grammars encoding linguists' knowledge of syntax, and that program was beaten, almost overnight, by statistical parsers trained on a corpus of example sentences (and later by neural models that used even less hand-designed structure). The painstaking linguistic knowledge did not stop being true; it stopped being the thing you coded.

Computational photography is living through exactly this. The hand-tuned denoisers, the edge-directed demosaickers, the sparse-gradient deblurring priors, the carefully sequenced ISP, each a substantial piece of engineering, and each now being overtaken, task by task, by learned and generative models. But photography has a twist that softens the bitterness, and it recurs throughout this book: the forward model is known. We understand the physics of how a scene becomes pixels through projection, optics, the sensor's noise, and the color-filter mosaic, so we can generate unlimited, perfectly-labeled training data by simulating that degradation, and we can build the physics into the model as an inductive bias. The domain knowledge does not disappear; it migrates from hand-coded algorithm into the data-generating process and the architecture's priors. So the right way to hold the field's two camps together is this: the bitter lesson says stop hand-coding the solution; computational photography's reply is but do put your hard-won physics into the data and the structure. Where each part of this book sits on that hand-designed → learned → generative axis is itself a thread we pick up explicitly in the Machine learning and Deep learning chapters.

1.1.5 Does photography still matter in the age of generative AI?🔗⧉

It is worth meeting the obvious provocation head-on. If a text-to-image model can conjure any picture you describe, such as a golden-retriever puppy on the moon, your living room redecorated, or a portrait in any style, why bother capturing anything? Is photography about to be a quaint craft, the way hand-lettering survived the printing press?

The answer this book backs is an emphatic yes, it still matters, more than ever, and the reason is what a photograph is. A photograph is indexical: it measures this light, from this place, at this instant. It records that this particular thing was there. A generative model produces something plausible; a camera produces something true. The value of a snapshot of your child's first steps is not that it is a well-composed image of a generic toddler, which a model could draw. It is that it is your child, that morning, those steps. No amount of plausible synthesis substitutes for the indexical link to a real moment, which is exactly why photographs serve as memory, as journalism, as legal and scientific evidence, as the record of a life. (That the link can now be forged is precisely why provenance and authentication have become urgent, a thread the book picks up in Authentication and Provenance (C2PA), but the forgeability of evidence is a reason to protect the camera's testimony, not to abandon it.)

The sheer scale says the same thing louder than any argument. Humanity now takes on the order of two trillion photographs a year, roughly sixty thousand every second, almost all of them on phones, a number that has only ever gone up. People are not photographing less as generative tools arrive; they are photographing more. And when they choose which phone to carry, the camera is consistently among the one or two features they care about most: imaging is the battleground on which flagship phones are marketed and won, which is why a phone maker will spend its scarce silicon, power, and engineering on a better ISP, a bigger sensor, and a smarter computational-photography stack. A device people buy largely to take pictures with is not a device for a dying art. The best camera, as the saying goes, is the one in your pocket (Figure 1.1.7).

The rest of the book uses a simpler framing: generative AI is being absorbed into photography. The same learned models that synthesize images from scratch now live inside the camera pipeline, denoising, deblurring, extending dynamic range, filling occlusions, upscaling, turning the photons a tiny sensor did capture into the photograph you wanted. Far from making computational photography obsolete, generative methods are its newest and most powerful tools, bolted onto a forward model we still understand. The camera's job, to measure the world, has not changed; what has changed is how much computation stands between the photons and the picture. That is the whole subject of this book.

1.1.6 A field at a crossroads🔗⧉

Computational photography sits at the crossroads of an unusual number of disciplines; few fields draw on this many at once. It draws on algorithms; AI and machine learning; physics (optics, light transport, radiometry); chemistry (film, sensors, dyes); electrical engineering (sensors, circuits, the image signal processor, or ISP); mechanical engineering (lenses, shutters, stabilization, camera bodies); computer graphics; computer vision; human perception; the art and craft of photography itself; human–computer interaction (HCI); performance engineering (real-time, on-device processing); and ethics.

And these do not take turns. They show up together, often inside a single feature. Take a phone's night mode. It leans on physics (catching scarce photons), electrical engineering (the sensor that catches them), algorithms (aligning and merging a burst of frames), AI (learned denoising), perception (tone-mapping the result so it reads naturally), performance engineering (finishing all of this in about a second, on a phone, without draining the battery), and ethics (how much to "beautify" a face, and whether to tell the user), all at once. That breadth is not a complication to tolerate; it is the appeal.

A few neighboring fields overlap with computational photography so heavily that it is worth saying exactly how they relate, and how they differ.

Computer vision is the closest. A useful way to carve up vision is by its three ingredients: 2-D images (the pixels themselves: filtering, features, matching), 3-D (the geometry behind the pixels: depth, shape, pose), and meaning (semantics: what objects and scenes the pixels depict). Computer vision ultimately cares most about the last two: it treats the image as a means to recover 3-D structure or to understand content. Computational photography inverts that emphasis: its product is the 2-D image, a better photograph. But it freely reaches into 3-D and meaning when they help the picture: depth to fake a shallow-focus background, segmentation to brighten a face and not the sky, recognition to pick the frame where everyone is smiling. The same tools, pointed at a different goal.

Computer graphics runs the arrow the other way, synthesizing images from models of geometry, materials, and light. Computational photography borrows its rendering and image-based-rendering machinery (and, increasingly, shares it outright: gradient-domain editing, light fields, differentiable and neural rendering all live in both). Image processing is the classical signal-processing bedrock the whole enterprise is built on: filtering, transforms, sampling, and compression. Computational photography is in large part image processing that knows where its pixels came from. And optics supplies the physics of image formation, which computational photography increasingly co-designs with the computation rather than treating as fixed (coded apertures, lensless cameras, end-to-end optimized lenses).

Because the field straddles these neighbors, so does its literature. A great deal of computational photography is published at the major computer-vision conferences — CVPR, ICCV, and ECCV — and at the premier graphics venue, SIGGRAPH (and SIGGRAPH Asia / ACM Transactions on Graphics). But the home of the community is ICCP, the IEEE International Conference on Computational Photography, the venue dedicated to the field itself.

1.1.7 It's not just about photography🔗⧉

We will talk about cameras and pictures throughout this book, because that is the most familiar and most vivid way in. But almost nothing here is really about snapshots. The underlying machinery, forming an image from measurements, modeling how light and optics and sensors behave, and inverting that model with computation, is the same wherever images are made, and once you have it you start seeing it everywhere.

Start with the ordinary camera pointed at extraordinary jobs. The instant a camera stops being aimed by a human enjoying the view and starts being aimed by a machine making a decision, the same pipeline serves a new master. Robotics and self-driving cars lean on cameras (and their cousins, depth sensors and lidar) to perceive, localize, and navigate. The alignment, depth, and motion estimation we develop for panoramas and refocusing are exactly what a robot needs to know where it is and what is moving. Surveillance and security, industrial inspection (spotting a hairline crack or a misplaced solder joint on a production line, where a camera that never blinks beats any human inspector), agriculture, drones, document scanning, and biometrics all use computational imaging for practical tasks. The picture is no longer the product; it is an input to a decision.

Then push the optics past what a handheld lens can do. Astronomy is computational photography at the grandest scale: telescopes stack thousands of frames to beat noise (the same averaging that powers a phone's night mode), use adaptive optics to cancel the atmosphere's blur in real time, and combine separate dishes into one synthetic aperture, the trick that let the Event Horizon Telescope assemble an Earth-sized virtual lens to image a black hole. At the opposite scale, microscopy does the same: light-sheet, confocal, and computational microscopes reconstruct 3D cellular structure, and super-resolution techniques reach past the diffraction limit we will meet in the optics part. Big or small, it is the same game: co-design the measurement and the reconstruction.

Change the wavelength and the field opens again. Visible light is a sliver of the electromagnetic spectrum, and imaging lives all along it. Infrared gives thermal cameras and night vision; ultraviolet reveals forgeries and surface detail; X-ray imaging, in the form of computed tomography (CT), reconstructs a 3D body from many 1D projections, a pure inverse problem; radio gives us radio astronomy and synthetic-aperture radar (SAR) that maps terrain through clouds and darkness. The whole of medical imaging is this same discipline under a different name: CT, magnetic resonance imaging (MRI), positron-emission tomography (PET), and ultrasound are all reconstruction problems, recovering an image from indirect measurements by solving exactly the kind of regularized inverse problem this book builds up to. A radiologist's scanner and a phone's night mode are closer cousins than either would guess.

And the techniques do not even need an image at all. Sampling, filtering, the Fourier transform, denoising, priors, inverse problems. The toolkit is really a toolkit for signals, and a signal can be anything. Audio is the obvious neighbor: a sound is a one-dimensional signal, and the same spectral and statistical ideas drive noise reduction, compression, and synthesis there. One example shows the connection: Paul Lamere's Infinite Jukebox takes any song, finds beats that sound nearly identical, and quietly jumps between them so the track plays forever without ever sounding wrong, which is precisely the idea of Video Textures (Schödl et al. 2000), a computational-video technique that finds similar frames in a short clip and loops between them to synthesize an endless, non-repeating video, transplanted from pictures to sound.

Run the trade the other way, too: point a camera at a quantity that has nothing to do with light and let computation read it out. Two of my favorite examples make the point. GelSight turns a camera into a sense of touch: press a slab of clear elastomer, with its outer face painted with an opaque membrane, against an object, and the gel molds to the surface's microscopic relief; a small camera behind the gel photographs that deformation under raking colored light and reconstructs a dense 3-D height map by photometric stereo. Because the optics, not a fingertip, set the resolution, it feels finer than human skin. It can read the ink of a banknote, the wear on a machined part, and the grain of a fingerprint, and it has become a workhorse for robotic manipulation, where a gripper needs to feel contact, slip, and texture, right down to vision-based tactile sensors built into the fingers of a dexterous robot hand Romero et al. 2024 (EyeSight Hand). Touch becomes an image, and every tool in this book applies to it Johnson & Adelson 2009. The second example points a camera at vibration: ordinary objects tremble by thousandths of a pixel in response to sound and force, and that motion, amplified and demodulated, turns a silent video into a measurement of the world's movement. A bag of chips behind soundproof glass becomes a microphone. The visual microphone recovers intelligible speech from the sub-pixel wobble of its surface Davis et al. 2014; the same sub-pixel motion, read as a modal signature, recovers an object's material properties Davis et al. 2015 and lets a single camera perform structural health monitoring of a bridge or a building, identifying its resonant frequencies and mode shapes from across the street [@chen-etal-2015-modal; @davis-2019-vibration]. We return to both threads in Revealing the invisible; here they are just the cleanest proof that "the image" was never the point. Touch and vibration are signals, and a camera plus computation is a remarkably general instrument for measuring them.

There is a deeper reason imaging keeps reappearing in places that have nothing obvious to do with pictures: at bottom, we really only know how to measure two things well: electrons and photons. Electrons are the workhorse, because nearly every sensor ultimately hands us a voltage or a current. But photons turn into electrons almost for free, through the photoelectric effect, in which a photon strikes a surface and frees an electron, the very phenomenon that won Einstein his Nobel Prize, and exactly how a photosite on a sensor works. Because that conversion is so cheap, a huge range of measurements is staged as the same trick: make the quantity you care about emit or modulate light, then count the photons as electrons. Fiber-optic and spectroscopic sensors, fluorescence assays, optical readout of pressure or strain or temperature, even reading the bits off a disc all work this way; scintillators turn invisible X-rays and gamma rays into little flashes of visible light that an ordinary camera can then photograph. The photon ⇄ electron conversion recurs because it is cheap, fast, and roughly linear, which is why the sensor we build up in this book, a photon counter at heart, is a template that reaches far beyond photography. The lesson worth carrying through the whole book is that you are not learning photography tricks; you are learning a way of thinking about measured signals that happens to be illustrated, beautifully, by photographs.

Because these connections are so many and so concrete, the book gives them a part of their own near the end, in Adjacent fields and applications, which tours imaging across disciplines and instruments: modern and exotic sensors, astronomy, X-ray imaging, medical imaging, microscopy, millimeter-wave imaging, sound and music, fluorescence imaging, opto-acoustic imaging, ultrasound, aerial imaging, computer vision, and robotics and self-driving. Each is the same machinery this book builds (measure, model, invert), applied in another domain; that part is where we follow the threads out.