1.4 What programming language for computational photography?🔗⧉

The ideas in this book are language-agnostic, but code has to be written in something, and the book ships in two editions, a Python one and a C++ one. We never show both in the same book; you pick the edition that matches what you're trying to do, because the two languages serve different moments. Python is for thinking and prototyping; C++ is for shipping and running fast. The concepts are identical either way. A third language, JavaScript, never appears in the code listings but is everywhere in the web edition: every interactive figure and live demo you can run in the browser is written in it (more below). The sections that follow are organized by where the code runs, because that is what really drives the choice: on the desktop (where you learn, prototype, and ship fast production code), in the browser (where the demos and this book's own figures are delivered), and on the phone (where computational photography reaches a billion people, under the tightest constraints of all).

The gap between those moments is larger than newcomers expect, and it is widest exactly when you write the low-level code yourself, say by implementing your own convolution rather than a library call. In Frédo's experience that naive pixel loop runs roughly 1000× faster moving from Python to optimized C++, and you can win at least another ~10× on top of that with Halide. So choosing a language is also choosing a performance regime, which is why the same algorithm can feel either instantaneous or hopeless depending on where you run it; we return to squeezing out those factors in Performance engineering and Halide.

1.4.1 On the desktop: Python, C++, and Halide🔗⧉

Python is the right default for learning, prototyping, and research. Its great virtue is readability: with NumPy, array code reads almost like the math it implements: out = a + 0.5b does what it looks like it does. It is fast to write and explore, with interactive notebooks and instant feedback, and it sits atop a vast ecosystem (NumPy, SciPy, Pillow and OpenCV, scikit-image, Matplotlib, and the entire deep-learning stack: PyTorch and JAX). Memory is managed for you, and it is the lingua franca of research and machine learning. The catch is speed at the pixel level: a naive per-pixel Python loop is orders of magnitude slower than C, so you must vectorize, push the loop down into NumPy's compiled core, or your code crawls. The global interpreter lock (GIL) limits true CPU threading, packaging and deployment are awkward, dynamic typing lets type bugs hide until run time, and you get little control over memory layout. Still, through NumPy and PyTorch it is fast enough* for most of what we do here.

C++ is the right default for performance and production. It is fast and predictable, close to the metal, with control over memory layout, cache behavior, and single-instruction-multiple-data (SIMD) vectorization, and it is what real camera ISPs, real-time pipelines, and shipped products are actually written in. Static types catch errors at compile time, and it deploys cleanly to phones and embedded hardware. The cost is real: it is verbose and slower to write, manual memory management is a rich source of bugs (out-of-bounds accesses, leaks, use-after-free), the build and toolchain story is heavier, and the edit–compile–run loop is far less interactive than edit–run. Once the algorithm is settled and it has to run fast, on device, at scale, that productivity tax is worth paying.

There's a third option for the hot paths. Halide is a domain-specific language, embedded in C++ and Python, for high-performance image processing. Its central idea is to separate the algorithm, what each pixel computes, from the schedule, how to tile, vectorize, parallelize, and fuse that computation across the memory hierarchy. You get near-hand-tuned speed without hand-writing the tangled, hardware-specific loops, and the same algorithm retargets to CPU, GPU, or DSP by swapping the schedule. It is the basis of real camera stacks. The cost is another language and mental model, best reserved for hot paths you have already identified. Premature scheduling is its own trap. We cover it properly in Performance engineering and Halide.

MATLAB deserves a historical note, because much of the computational-photography literature was prototyped in it. For years it was the platform for this kind of work: a strong numerical and linear-algebra library, and an interactive IDE that could read, display, and manipulate images natively: an image was just a matrix, imread/imshow and array slicing made experiments effortless, and the Image Processing Toolbox supplied the rest. It has since been largely displaced by Python (NumPy fills the same array-is-an-image niche, free and open) and especially by PyTorch, whose automatic differentiation and GPU / deep-learning ecosystem MATLAB never matched. We mention it because older papers and code you will meet are written in it, and the array-thinking it taught carries over directly to NumPy.

1.4.2 In the browser: JavaScript and the web stack🔗⧉

JavaScript is the language of delivery in the browser, and it earns its place here because it is what powers this book's own interactive figures and demos: the BRDF sphere, the rainbow drop, the ray-tracer, the live HDR merge, chroma-key matting, the camera-fed panorama stitcher, the 3D camera museum on the cover. Every one runs entirely in your browser, no server, on whatever phone or laptop you happen to hold. That reach is the point: JavaScript is the only language that runs natively everywhere, and the modern browser ships a surprisingly complete imaging stack for free. The Canvas 2D API hands you raw pixels (getImageData / putImageData expose an RGBA Uint8ClampedArray you can poke per pixel), which is all you need for filtering, point operations, and small images. For anything real-time or full-resolution you reach for the GPU: WebGL / WebGL2 (and the newer WebGPU) run pixel work as parallel shaders, the only way to get smooth interaction at full frame size. getUserMedia opens the live camera (used in several demos here), typed arrays (Float32Array and friends) give tight numeric buffers, and Web Workers plus OffscreenCanvas move heavy work off the UI thread. Modern just-in-time engines make a hand-written pixel loop far faster than the same loop in pure Python, though still well short of C++ with SIMD, which is exactly what WebAssembly (WASM) is for: compile C/C++/Rust image code and run it at near-native speed in the page, the bridge straight back to the performance languages above.

The library landscape mirrors the others. For 3D and GPU rendering, three.js is the de-facto WebGL engine (this book's 3D optics and cover scenes are built on it), with regl or twgl for lower-level control and gl-matrix for vector/matrix math. For classical vision, OpenCV.js is OpenCV itself compiled to WASM, the same catalog, in the browser. For anything learned, TensorFlow.js and ONNX Runtime Web run neural nets client-side (on the GPU via WebGL/WebGPU), and Google's MediaPipe offers ready-made real-time face / hand / pose tracking (the live face-landmarks demo uses it). The rule of thumb from the Python and C++ worlds holds unchanged: implement the core once to understand it, then stand on these to build.

A handful of things bite web image code specifically, and are useful to know up front. Where the pixels live matters most: a per-pixel JavaScript loop over ImageData is fine for a thumbnail but stalls on a full-resolution photo. Push that work to a shader, a Worker, or WASM. Color and gamma: canvas pixels are 8-bit, sRGB-encoded by default, so do your arithmetic in linear light and respect the gamma (a recurring lesson of this whole book); float and wide-gamut canvas modes exist but are not yet universal, and 8-bit RGBA loses precision fast across a multi-step pipeline, so keep intermediates in Float32 buffers or float textures. Security: drawing a cross-origin image without the right CORS headers taints the canvas and silently blocks getImageData. Serve your assets same-origin or with CORS. The camera needs a secure context: getUserMedia works only over HTTPS (or localhost) and with the user's permission, and facingMode picks the front or rear lens on a phone. And high-DPI displays: scale the canvas backing store by devicePixelRatio, or everything you draw looks soft on a retina screen.

1.4.3 On the phone: Android and iOS🔗⧉

On the phone is where computational photography most often actually ships (the night mode, the portrait blur, the multi-frame HDR in a billion pockets), and also where it is most constrained and most fragmented. It helps to split a camera app into four layers (capture, processing, ML, and UI) because the two ecosystems diverge most at capture and least at the processing core. On iOS you write Swift (SwiftUI or UIKit for the shell). Capture is AVFoundation (AVCaptureSession / AVCapturePhotoOutput, with manual exposure / ISO / focus and Bayer RAW or ProRAW, a DNG that encodes Apple's computational result as a gain map plus semantic data, though you can still request the sensor RAW to bypass it). Processing is Core Image filter graphs for the productivity path, or Metal / Metal Performance Shaders when you need custom kernels (with vImage / Accelerate for CPU SIMD); ML is Core ML plus turnkey Vision. The iOS stack is coherent and hands you a great deal for free, at the cost of an opaque-by-default pipeline you sometimes have to fight to get under. On Android you write Kotlin, and capture is the awkward part: camera2 is the low-level, verbose, full-manual interface (the productized descendant of the Frankencamera — RAW / DNG via DngCreator), while CameraX is the far more pleasant Jetpack wrapper that papers over device fragmentation (with camera2 interop and vendor night / bokeh / HDR Extensions). Android's processing story is weaker (there is no Core Image equivalent and RenderScript is dead), so you reach for Vulkan compute (the forward path), legacy OpenGL ES, or native C++ with NEON; ML is LiteRT (the rebranded TensorFlow Lite) with GPU and vendor delegates, since NNAPI is being wound down.

The professional answer for a both-platforms app is to stop duplicating the hard part: write the imaging pipeline once in C++ and bridge it, JNI on Android, Swift's direct C++ interop (or a thin Objective-C++ shim) on iOS, so one correctness-bearing core (LibRaw to decode raw, OpenCV, Eigen, and Halide for the hand-tuned-but-portable stages) drives two thin platform skins, with only the GPU-bound stages resolving to Metal on iOS and Vulkan on Android. Cross-platform frameworks (Flutter, React Native) are fine for the app shell but own only the UI. They spare you neither AVFoundation / camera2 nor the shared core. The architecture choice to settle early is how much of each vendor's computational pipeline you inherit versus replace: the marquee pipelines (Google's HDR+, Apple's Deep Fusion and Night mode) are largely closed, so if your app's value is your own pipeline you architect around grabbing the rawest frames each platform will give you and doing the rest yourself. Either way the phone is a heterogeneous, power- and thermal-limited computer (big.LITTLE CPU cores, a mobile GPU, the ISP, an NPU, all under a battery budget), which is exactly why real phone stacks lean on Halide: separate the algorithm from a per-target schedule and retarget the same code to CPU, GPU, or DSP. The Systems part returns to these APIs in detail in Photography programming on phones, where the on-device-versus-cloud question, where the computation should run at all, is taken up as well.

1.4.4 Libraries🔗⧉

This book has you implement the core algorithms from scratch, for example a bilateral filter, a homography solve, or a Poisson blend, because writing one yourself is how you actually understand it (the reference implementations in imageops exist for exactly that). But once a technique is clear and you want to build with it, or dive straight into an advanced project, reach for mature libraries instead of reinventing the plumbing. In Python, the workhorses are Pillow (PIL) for loading, saving, and basic operations, NumPy / SciPy for array math, OpenCV and scikit-image for a large catalog of vision and image routines, and PyTorch for anything learned (and as a GPU array + autodiff engine even when no network is involved). In C++, the counterparts are OpenCV again (the de-facto standard), Eigen for linear algebra, LibRaw for decoding camera raw, and Halide for the performance-critical pipelines above. In the browser, the same ground is covered by OpenCV.js, three.js, and TensorFlow.js (see the JavaScript discussion above). The rule of thumb: implement once to learn, then stand on the libraries to go far.

1.4.5 Vibe coding🔗⧉

A note on vibe coding, writing image code with the help of a large language model (LLM). It is now genuinely part of the toolkit: great for boilerplate, file I/O, visualization, and explaining a baffling error, and a quick way to draft a function or a test. But image code is numerically and perceptually subtle, and "looks plausible" is emphatically not "correct." A resampling kernel that is subtly wrong, a color conversion off by a gamma, an antialiasing filter that rings. These produce images that look fine until you check them. So keep the model on a short leash for filtering, resampling, color, and antialiasing, and always verify against a hand-checkable input, a constant, an impulse, a half-plane, before you trust it. Treat generated image code as guilty until tested (see Developing, Testing and Debugging).

And don't assume the failures will be confined to the hard parts. The recurring surprise, building this book, was the opposite: the coding agent would nail the genuinely difficult algorithm and then fumble the trivial step sitting right on top of it. The clearest case came while reproducing Photobios, the morphing slideshow that ages a person across a whole photo collection (Many images and photo collections). The model handled the hard parts on its own, registering the faces from facial landmarks and finding a dynamic-programming path through the collection so consecutive frames matched, and then repeatedly botched the easy part, a one-line cross-fade, frame = (1−t)·start + t·end, the linear blend you would write in your sleep. It took something like five re-prompts to get that single line right. Competence on the hard stuff is no guarantee on the easy stuff, so review even the trivial parts. The full version of this anecdote lives in Developing, Testing and Debugging.

One practical point supports this claim: every interactive figure and live demo in this book was itself vibe-coded, and many of them truly run the real algorithm in your browser. The shift-and-add light-field refocus, the HDR merge, the panorama stitcher, the bilateral grid, and the Poisson blend are not canned videos but the actual computation, executing on your device as you drag the sliders. That this was feasible at all, for one author, is the endorsement. But it was rarely first-try: in many cases the working demo took several rounds of debugging between the author and the coding agent, with the author spotting that a result looked subtly wrong (a halo, a color cast, a fade gone dark), the agent proposing a fix, and back and forth until it was actually correct. Which is the whole point of this section: the agent is a fast, fallible collaborator, and the human's job is to know what correct looks like and to keep checking until the pixels agree.