3.8 Programmability, or the lack thereof🔗⧉

Here is an obstacle that shapes this entire book. Most cameras run closed firmware: you cannot run your own code on them. The computational-photography ideas in the chapters ahead, including multi-frame high-dynamic-range merging, burst denoising, and computational bokeh, all need programmable capture: control of the per-frame exposure, focus, and gain, and access to the raw burst before the camera's own pipeline mangles it. A stock camera grants none of this, so its computational ideas cannot even be tried on most cameras (Figure 3.8.1).

The community responded with hacks: CHDK (Canon Hack Development Kit, for PowerShot compacts; CHDK (Canon Hack Development Kit)) and Magic Lantern (for Canon DSLRs; Magic Lantern) are add-on firmware that expose scripting, raw video, auto-bracketing, intervalometers, and focus stacking. Those features exist because the manufacturers withheld them. Phones are the more open platform: Android's Camera2 / CameraX API (and Google's earlier FCam research camera) expose per-frame control of exposure, focus, and gain, plus access to the raw burst. That openness is why phones, not cameras, became the home of computational photography. The platform that grants programmable capture is the platform where the research happens, and it is the bridge to the rest of this book.

There is also a maker path: a Raspberry Pi with a camera module (the HQ camera, driven by the libcamera / picamera2 stack) is a cheap, fully open, fully programmable camera. It hands you per-frame control of exposure and gain and direct access to the raw Bayer frames on a hobbyist board you can script in a few lines of Python. That makes it the natural platform on which to try this book's algorithms and build a do-it-yourself computational camera.

Phones differ from cameras in access and raw compute. A modern phone is a small supercomputer wrapped around a sensor: even a low-end handset carries a multi-core CPU, a GPU, a dedicated image-signal processor (ISP), and a neural accelerator (NPU) of a few trillion operations per second (TOPS); a high-end phone SoC pushes the NPU into the tens of TOPS (roughly 30–50), enough to run several neural networks per frame at capture time. That is exactly the budget computational photography spends: burst alignment and merging, learned denoising, semantic segmentation for portrait mode, real-time HDR, and it is why the techniques bloom on phones. A dedicated camera, by contrast, spends its silicon on a fast, fixed-function ISP (Canon's DIGIC, Sony's BIONZ, Nikon's EXPEED) tuned to push high-resolution raw through a conventional pipeline at high frame rates, with comparatively little general-purpose or neural compute and, until very recently, no programmable on-device ML at all. The contrast is clear: the device with the better optics and bigger sensor has the weaker, more locked-down processor, while the device with the tiny sensor has the compute and openness to compensate for its hardware limits. Most of this book is about spending that second kind of advantage.