☰ ‹
draft · v0.1.240 INTRO From Digital to Computational Photography Computation is the new optics The goals of computational photography What makes a technique useful A field in transition: from hand designed algorithms to learning to generative AI Does photography still matter in the age of generative AI? A field at a crossroads It's not just about photography How to read this book Machine-readable by design How this book was made What programming language for computational photography? On the desktop: Python, C++, and Halide In the browser: JavaScript and the web stack On the phone: Android and iOS Libraries Vibe coding Programmability, or the lack thereof Problem sets FUNDAMENTALS OF IMAGING Light and optics Light: rays, waves, and the spectrum How light is created Light power and brightness Summing coherent vs. incoherent light Reflection, refraction, and what happens at a surface Light-matter interaction is largely resonance Polarization The color of objects: illumination times reflectance The BRDF and the look of materials Illuminants Radiometry: radiance, irradiance, exposure, and falloff Global illumination Wave effects, diffraction, and the diffraction limit Pinhole image formation and linear perspective Pinhole imaging and the perspective projection Homogeneous coordinates Camera in a general configuration Intrinsics, extrinsics, and what cropping really does What perspective preserves, and what it destroys Wide-angle distortion: spheres bulge and faces stretch at the edges Photography with focal length: framing, magnification, and compression Depth, ray length, and unprojection Lens image formation Single lenses: refraction and Snell's law Thin lens optics Aperture and the f-number Image measurements as integrals The plenoptic function The pixel integral Exposure values (EV) and stops Splitting the integral Depth of field The circle of confusion Depth of field versus depth of focus Choosing the maximum circle of confusion The double cone, and the near and far limits Hyperfocal distance The surprising invariance Background blur beyond the focus plane Defocus is not a blur of the image Sensor size scales depth of field Motion blur Blur from subject motion Blur from camera shake How shift-invariant is camera-shake blur? Which dominates: translation or rotation? The hand-holding rule of thumb Shake from the shutter and mirror Panning: fighting subject motion by moving the camera Freezing motion with light Sensors: photosites, CCD vs CMOS The photoelectric effect Photon to number: the photosite Beyond the visible spectrum: near-infrared, thermal, and ultraviolet Analog-to-digital conversion CCD versus CMOS Time integration: the exposure Shutters A taste of modern sensor tricks The impact of sensor size How far sensors have come Noise, signal-to-noise ratio and dynamic range Noise sources The algebra of noise Signal-to-noise ratio: it's about ratios Noise variance is affine in pixel brightness Noise as a function of ISO Dynamic range Regimes, and how close we are to the limits Imaging as a linear system Linear systems from a billboard to an image Linear systems from ray space (light fields) to an image Space–time: imaging across time When linearity breaks down Where this is going: invertibility Imaging as an inverse problem Linear algebra: how hard is this problem? Priors and the manifold of natural images Probability: the Bayesian view Optimization, inference, loss Deep learning, amortized inference, and generative AI When you can design the operator — and a teaser of information theory Recap: what makes imaging hard, and where it's going Harder inverse problems: factorization VISUAL PERCEPTION AND COLOR Anatomy of the eye and image formation Perceptual color and trichromatic vision Cone Photoreceptors and Perceptual Color Metamers Color blindness Opponent process and the multistage model So what are the primary colors? Color as linear algebra Measuring and encoding color analysis vs synthesis and non-orthogonality Measuring color Chromaticity diagram Linear vs Gamma vs. log encoding RGB color spaces Non-linear perceptually-uniform color space HSV, HSL, and cylindrical color spaces Reproducing color Skin tones Sensing color: multiplexing strategies Multiplexing in time Multiplexing in space: the color filter array Multiplexing across separate sensors: the beam-splitter Multiplexing per pixel by dispersion: color routers and nano-prisms Multiplexing per pixel by tunable absorbers: quantum dots Multiplexing in depth: stacked photodiodes Hybrid strategies Beyond trichromatic capture: full spectrum and multispectral Spectral sensitivity: matching the eye, catching photons, and white balance Visual processing Light adaptation Lightness constancy Color constancy Contrast Spatial vision Temporal vision Attention and eye movements The visual system makes things up Animal eyes The optics: many ways to form an image The pupil: aperture shape as a clue to ecology Amphibious vision: focusing in air and in water The sensor: a retina that is not uniform Seeing in the dark: sensitivity and its tricks Acuity: the finest detail, and how to sharpen it Color vision: opsins remixed Polarization: the axis we are blind to Temporal vision: flicker, and motion first PHOTOGRAPHY Exposure Basic photography: exposure settings — shutter, aperture, and ISO Metering: from printed guides to the sensor f-stops versus T-stops From exposure to photometry Exposure modes, UI, and auto-ISO Lenses Lenses and focal length Aperture and the diaphragm Focusing the lens Image stabilization Keeping the lens clean Lens filters: polarizers, ND, and graduated ND Focus, autofocus, and depth of field Manual-focus aids: making the focal plane visible Cameras UI and display How the viewfinder works Anatomy of a modern full-frame interchangeable-lens camera Anatomy of cell-phone cameras Camera versus phone Types of cameras Cameras beyond photography A camera's other sensors Video Cinema lenses Illumination and the flash Natural illumination Flash Flash metering Indirect illumination and bounce flash Multiple-point lighting Traditional and Digital Darkroom The traditional darkroom The digital darkroom Limitations of the medium Compensation, accentuation, conflict The picture is flat: depth and its cues The picture is static: time and motion One viewpoint, and a finite frame The contrast is limited: dynamic range and gamut Resolution: the limitation we are overcoming Displays The range of displays Display technology HDR displays and dual modulation The film movie projector Printing Why the display characteristics matter Distance, resolution and acuity Sharpening for the display: size and distance Light level viewing conditions Robustness of perspective to the viewer's viewpoint Gamut and gamut mapping Color management, ICC, and industry standards Photographs are usually not passive objective recordings Faithful is not the same as realistic Choosing the instant, choosing the viewpoint Speed, aperture, and focus: the photographer rules Filters and lighting: building the scene's light The darkroom: tone, dodging, and burning Make-up and retouching: authoring the subject Getting closer — systematic capture, and the biases that remain Types of photography People Places Animals Motion and the decisive instant Objects and the arranged frame Low light and the sky Art and abstraction Technique and apparatus One frame, or a body of work Photography and videography jobs The market: what the image is for The role: where in the pipeline The employment model A job is a triple How the opportunities have shifted The crew, collapsing toward the individual Why people take photos A small map of motives From a memory technology to a communication one What the psychology actually shows When people take photos, and why the usual numbers are not evidence Why this belongs in a book about how images are made BASIC IMAGE PROCESSING AND ISP Image representation An image is an array Float vs 8-bit vs more bits Stochastic Quantization and Dithering How the array sits in memory Pixel coordinate conventions Alpha and extra channels Video and more dimensions Pixel lookups and boundaries Beyond the pixels: basic metadata and EXIF Three kinds of operation What the numbers mean What the numbers actually mean: encoding What the numbers actually mean: color spaces Developing, Testing and Debugging The workflow: build, run, look at the picture Principles Test on inputs you can verify by hand Finding good real test inputs Per-algorithm debug recipes Crashes and bounds Vibe coding: writing image code with an LLM Point operations Three kinds of operation The point-operation curve Exposure: a multiply in linear light Brightness vs. exposure Contrast: steepen about a pivot Black point, white point, and levels The general tone curve Basic color enhancement: saturation and vibrance Converting to black and white Lookup tables Video color grading Local point operations Histograms the histogram the histogram depends on the encoding space a histogram is a sampled estimate histogram equalization histogram matching constraining the slope: Ward's histogram-based tone mapping Global tone mapping The dynamic-range problem Two families: global and local Global tone mapping: one curve for everything Histogram equalization and Ward's bound Local tone mapping and the halo problem: why naive local fails Why log space pays off The analog ancestor: the Zone System HDR on every screen: the gain map Where this is going Beauty curves: the camera "look" Why scene-linear looks dull The shape of the curve Highlights: roll off to white, not to a color Saturation, and the rest of the "look" The discipline Neighborhood operations and convolution Motivation: blur and sharpen Convolution 101 The flip: where-from vs. where-to Properties: the impulse, normalization, symmetry, commutativity Nitty-gritty: finite images A blur zoo Separability Gradient and oriented filters Is imaging blur usually a shift-invariant convolution? Where this goes next Sharpening Linear sharpening Why it's called "unsharp mask" Sharpen in gamma, linear, or log? Non-linear sharpening Fourier Images as vectors in a high-dimensional space why Fourier Definition: one coefficient per wave Sines are the eigenvectors of convolution The 2-pixel example Reading an image's Fourier transform A small bestiary of transforms The fine print: two limitations of Fourier Windowing: the price of measuring a real spectrum Application preview: can we deblur, given the blur? Sampling, downsampling, and aliasing Aliasing: a high frequency in disguise Nyquist and the sampling theorem In two dimensions, aliasing has a direction Sampling in the frequency domain: spectral replicas Seeing it: sampling and pre-filtering, hands on The ideal reconstruction filter: sinc, and why it is unreachable Resampling and upsampling Domain operations: moving pixels around Start with scaling up The naive idea, and why it leaves gaps Loop over the output, use the inverse transform Nearest neighbor Linear interpolation, in 1-D first Bilinear, in 2-D The convolution perspective Better kernels: bicubic and Lanczos Upsampling vs. downsampling: scale the kernel Prefiltering when the transform isn't a clean scale Beyond Nyquist-limited reconstruction Where this goes next Pyramids and wavelets Halfway between space and frequency The idea: process an image at many resolutions The Gaussian pyramid The Laplacian pyramid Reconstruction: an exact encoder and decoder Each level is a frequency band What the bands look like on a real image Candid limitations Cousins: wavelets and steerable pyramids Pyramid blending Other applications Multiresolution as a recurring theme Image metrics Full-reference vs. no-reference why not just L2? PSNR SSIM VDP and HDR-VDP learned metrics The right metric depends on the task Denoising basics what is noise? denoising by averaging multiple shots denoising from a single image Spatial averaging and its limits The bilateral filter: averaging by affinity Denoising in the pyramid: coring Denoise color more than brightness noise estimation The limits of denoising Demosaicking Reminder: the Bayer mosaic The task: full RGB at every pixel The naive approach: interpolate each channel on its own Why naive interpolation zippers: averaging across an edge Doing better: edge-directed interpolation The harder half: red and blue, and color fringing Green-based demosaicking: interpolate the color difference Classic (non-learning) demosaicking: the general strategy Related: the optical anti-aliasing filter Quad-Bayer sensors: remosaic before demosaicking Fuji X-Trans: a larger, irregular CFA Beyond hand-tuned: joint denoising and learned demosaicking Cross-reference: other ways to sense color Where this sits in the pipeline Auto-exposure and auto white balance Auto-exposure: metering white balance and color constancy Automatic white balance The limits of white balance, and CRI File formats and compression The big picture: none, lossless, lossy Data versus metadata: EXIF PNG: the format we read from JPEG: compression by perception RAW files: before the cooking HDR formats: more than 8 bits Modern formats Other formats: TIFF and GIF Recap ISP, non-destructive editing: A basic ISP Pipeline design and tuning The ISP, evolving: traditional → learned → generative Recap 2: non-destructive editing COMPUTATIONAL TOOLSLinear Inverse Problems and RegressionBlur is linear, so deblurring is inversionImages as vectors, and the notation overloadRegression: deblurring as least squaresMatrices without forming matrices: gradient descent and conjugate gradientEfficient solversFundamental limits of inverse problemsThe question: what can no algorithm recover?The SVD picture: null space, near-null, and the noise explosionInformation bounds: a floor no estimator beatsThe resolution–noise tradeoff, and how a prior inventsFundamental versus practical, and coded capture as the escapeInformation theoryThe basics, at a high levelCompression: the clearest applicationThe imaging system as a channelInformation-theoretic quantities as objectivesThe modern revival: learned compression and generative modelsCousins, not childrenWhere it is least relevant: aesthetics and perceptionMachine learningThe framing: learned operators replace hand-designed onesThe data story: synthetic data, noise models, datasetsDeep learningLow-level learned operators (pixel-to-pixel)Mid- and high-level learned predictorsGenerative models: image-to-image translationLearned perceptual metrics and lossesGenerative AI and diffusionThe framing: generation is learning and sampling a prior $p(x)$Diffusion: generation as iterated denoisingConditioning: text, images, and controlPosterior sampling: generative priors for inverse problemsOther generative families, in briefCaveats and ethicsImage priorsA catalog of priors: from smoothness to learnedShift and scale invarianceAnisotropy: sparsity in a single directionLinear color modelsDenoising as a universal prior: Plug-and-Play and REDDiffusion is iterated denoising (the continuous limit)Score-based models, the thin manifold, and imperfect priorsEDGES MATTERBilateral filteringMotivation: local tone-mapping haloesBilateral filterCross / joint bilateralBilateral gridBilateral-grid learning (HDRnet)Non-local meansEdge-preserving optimization — colorizationThe shift: filtering becomes optimizationColorization: the canonical demonstrationWriting down the energySolving it: a sparse linear system we already ownOne affinity, two faces: the matting Laplacian connectionLocal Laplacian filtersThe wish, and why the bilateral leaves a haloWhat a Laplacian pyramid hands youThe mechanism: a remapping recomputed per output pixelOne curve, three edits: enhancement, tone mapping, inverse tone mappingContrast with the bilateral base/detail splitThe cost, and the fast versionMatting LaplacianThe matting problem: compositing, and why it is under-determinedThe color-line assumption: α becomes linear in colorClosed-form matting: the matting LaplacianSpectral matting: eigenvectors of the matting LaplacianAn alternative matting LaplacianBeyond matting: spatially-varying white balanceGuided image filteringThe key idea: the output is a local line in the guideFitting the line, and the one knobWhy it is fast: O(N) regardless of window sizeNo gradient reversalReading the affinity back outWhere it is usedLocally adaptive regression kernel (LARK)Change the verb: from averaging to fittingSteering the kernel to the local structureThe regression flavor of the affinityLARK as a structure descriptorPoisson image editingEdit gradients, not pixelsSeamless cloning: the headline applicationReconstruction = solving the Poisson equationPoisson vs. pyramid blendingAdvanced techniquesSeam optimizationFind the non-edges: least-noticeable cutsIntelligent scissors / live-wire: the cut as a least-cost pathSnakes (active contours): the continuous cousinGraph cut: globally optimal boundaries by min-cut / max-flowGrabCut: interactive cutout from a single rectangleNormalized cuts: the spectral relaxationSeam carving: content-aware resize by DPTime-lapse via DP: a seam through timeVideo textures: looping a seam in timeGraphCut textures & photomontage: cut, then blendWhere this connects: MRFs and beyondRecap: which edge-aware technique when?The three relationships to edgesPros, cons, and cost: when to reach for eachA worked chooserWARPING, MORPHING, AND LAGRANGIAN APPROACHESWarpingWarping is a domain transform: move where, not whatForward vs inverse warping: why output-driven winsResampling: a quick reminder (developed in BASIC)Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladderSpecifying the warp II: free-form warps from sparse correspondencesLiquify: the warp with a paintbrush bolted onAntialiasing for complex transformsWhen the footprint stops being a square: the warp JacobianEWA: the elliptical weighted averageFeline and anisotropic MIP probing: EWA on a GPU budgetOther footprint integrators: summed-area tables and ripmapsFootprint estimation, clamping, and when to reach for itMorphingWhy a cross-dissolve isn't enough: the ghosting motivationTwo interpolations: domain (shape) and range (color)The morphing recipe (combine both)Field morphing: the Beier–Neely line-pair warpMesh / triangulation morphing (the alternative warp)View morphing: the geometrically-correct in-between of two viewsRecap and significanceMorphable modelsStep 1: dense correspondence, the precondition for averaging shapesStep 2: Procrustes alignment, factor out pose so PCA sees shapeStep 3: PCA, a mean and a basis of eigen-deformationsStep 4: shape and appearance are separate basesStep 5: fitting a new photo, analysis-by-synthesisStep 6: once fitted, edit by moving in the spaceStep 7: from linear PCA to neural priorsRecap and significanceShape-preserving warpingWhy plain interpolation shears: the rigidity gapAs-rigid-as-possible (ARAP) manipulationLinear blend skinning and why the weights decide everythingBounded biharmonic weights (BBW)ARAP interpolation: rigid-as-possible morphing between two posesMoving least squares: the simpler cousinRecap and significancePerspective distortion and its correctionKeystoning is projection, not a lens flawThe fix is a homography: re-render the façade fronto-parallelThe optical alternative at capture: tilt-shift / ScheimpflugCropping wide-angle photographs: recenter with a homography, don't just cropA different perspective distortion: wide-angle portraits, and a content-aware fixBeyond faces: correcting arbitrary objects (MaDCoW)A montage has no single viewpoint: a family in a boxRecomposing perspective after capture: computational zoomThe catch: resampling cost, and "only a plane rectifies exactly"Where this sits: one map, then transportMATCHING PIXELS AND HUMANS ACROSS SPACE AND TIMEBrute forceWhy you must align firstBrute-force translational alignment (SSD / NCC)Phase correlation: the whole shift from one FFTCoarse-to-fine alignment on a pyramidSub-pixel matchingRefining the cost: fit a parabolaRefining in the frequency domain: phase correlationRefining by gradient: the Lucas–Kanade stepWhat breaks it: peak-locking, texture, and noiseOne tool, everywhere downstreamSparse matchingWhere to look: corners, and the structure tensorInvariance: surviving scale and rotationDescribing a neighborhood: SIFT and its zooMatching: nearest neighbor in descriptor spaceFeature trackingTracking vs dense flow: sparse-but-long vs dense-but-shortKLT = Lucas–Kanade, per feature, iterated over timeGood features to track = where the structure tensor is well-conditionedWhat breaks long-term tracking: drift, appearance change, occlusion, and re-detectionModern point trackers (briefly)An application: synthetic motion blur from a trackRobustness: the ratio test and RANSACThe ratio test: reject ambiguous matches before fittingRANSAC: fit from a minimal random sample, score by consensusVariants and degeneracyDeep learning approaches to sparse matchingLearned local features: SIFT's job, done by a networkLearned matching — reason about the whole set at onceThe 3-D-aware turn: pointmaps subsume matchingFast matchingApproximate nearest neighbors for sparse descriptorsPatchMatch: randomized dense correspondence by propagationFast high-dimensional matching by random projectionOptical flowWhat optical flow is: and whether it is even well-definedBrightness constancy and the optical-flow constraintThe aperture problemLucas–Kanade: local constant-flow least squares (and the structure tensor)Horn–Schunck: global smoothness regularizationLarge motion: coarse-to-fine warpingLearned flow: RAFT (neuralize the classical pipeline)Deep learning approaches to optical flowThe unrolling principle: neuralize the classical solverCost volumes and warping inside the netRAFT and the recurrent updateFace trackingDetecting the faceLandmarks: pinning down the featuresTracking across timeLifting to 3-D: the morphable modelRecognition, and the dark sideWhich library to useBody pose estimationTop-down vs bottom-upOn-device, real timeLifting to 3-D: parametric bodiesWhich library to useSINGLE IMAGE COMPUTATIONAL PHOTOGRAPHYDenoisingA reminder: bilateral filtering and wavelet/pyramid shrinkageBM3D: group similar patches and filter them togetherLearning to denoiseSuper-resolutionWhat problem super-resolution solves (and why it's ill-posed)Scenarios: single-image, burst, and hybrid space–timeReconstruction vs hallucination: measured detail vs invented detailDemosaicking and joint reconstructionThe pipeline problemFlexISP: one energy, one priorLearning the joint prior: the founding formulationModel-based and unrolled networksLearning where the hard cases areCo-designing the mosaic: a learnable sensorThe end-to-end learned ISPBetter backbones: CNN → transformer → diffusion priorExotic mosaics: learned remosaickingWhere do the labels come from?Correcting chromatic aberration jointlyRecapNon-blind deblurringDeblurring in the presence of noise: why naive inversion failsThe Wiener filter — the regularized, noise-aware inverseSparse gradients: the prior that keeps edges sharpBlind deblurringBlind deblurring: estimating the kernel and the imageA more realistic blur model: spatially-varying (camera-shake) blurEngineering the aperture: depth and all-focus from a coded maskDehazingDehazing as a prior-driven inverse problemDifferentiable image pipelines and algorithm optimization (Halide)Mixed-lighting white balanceWhy a global gain must failEstimating the per-pixel mixture: an under-determined inverse problemCorrecting each region for its own lightWhat came afterInpainting, texture synthesis, and object removalInpainting as filling unmeasured pixels — the spectrum of priorsPDE / diffusion-based inpaintingTexture synthesis (Efros–Leung; Efros–Freeman quilting)Exemplar inpainting: clone, healing brush, and object removal (Criminisi)Data-driven scene completion (Hays & Efros)Deep inpainting (context encoders → partial/gated conv → diffusion)Highlight / specular recoveryEpitomes — a compact patch modelPatch matchThe nearest-neighbor field, and why exhaustive search is the bottleneckPatchMatch — randomized correspondence Barnes et al. 2009Applications: hole filling, retargeting, reshuffleShift-Map image editing Pritch et al. 2009 — editing as graph-cut labelingColorizationOne channel in, three channels outA spectrum of priors: scribbles, references, and learned modelsThe multimodality trap: why naïve colorization goes muddyClosing the loop: learned priors with a human's hintsPlausible is not correctWhere it sitsCompositing, segmentation and mattingCompositing and alpha channelsSegmentation: cutting the object outThe fundamental matting equationBlue/green-screen matting and chroma keyingTraditional matting approachesDeep-learning mattingGenerative mattingHarmonization and where the blends liveOptical effects beyond alphaIllumination related effects in a single imageIntrinsic images: the unifying frame ($I = R\cdot S$)Multiple-light / spatially-varying white balanceReflection removal — pulling apart a transmission and a glass reflectionShadow detection and shadow removalSpecular-highlight removal / "fake polarization" (the dichromatic model)Tone MappingGlobal vs local, re-hashedA smarter global curve: histogram adjustmentA taxonomy of local methodsThe darkroom ancestor: dodge & burn and the Zone SystemStyle transferClassical style transfer: patches and statisticsNeural style and feed-forward stylizationStyle transfer as image-to-image translationNon-photorealistic renderingWhat NPR is for, and the one ideaStroke-based / painterly rendering (and the brush p-set)Edge-preserving abstraction: bilateral + Difference-of-GaussiansExample-based stylization and the bridge to neural styleRegion-based stylization: stained glass, low-poly, mosaicsArtistic screening and halftoningCOMPOUND LENSES, AND ABERRATION CORRECTIONAberrations and optical challengesTaxonomy of challengesSpherical aberrationComaAstigmatismField curvatureChromatic aberrationRadial distortionWave effects and diffractionVignettingFlare and coatingAberrations correctionThe two families of cureCorrection in glassComputational correctionRadial distortion correctionMeasuring lens qualityThe Modulation Transfer Function (MTF)Measuring MTF in practiceSpot diagrams and the PSFMTF through focus and depth of focusField-dependent MTF: sagittal vs tangentialColor, geometry, and illumination measurementsScalar quality summariesLens optimizationThe high-level idea: design as optimizationThe forward model: ray-tracing and spot diagramsFrom hand calculation to softwareTradeoffs: the design is always a compromiseTolerancing: from the nominal design to a manufacturable oneA short bestiary of classic designsThe lens as a system: cardinal points, pupils, f-number, T-stopScaling laws in opticsLohmann's scaling laws: why a good lens is heavyThe gigapixel barrier for a single lensThe escape: monocentric multiscale opticsCapture everything, crop later: the spatial cousin of the light fieldSpecial opticsTilt-shift and the Scheimpflug principleFisheye and non-rectilinear projectionMirrors: catadioptric and reflecting systemsPeriscope / folded-lens design (smartphone telephoto)Anamorphic opticsStereo (3D) lensesMacro, microscope objectives, and telescopesTeleconvertersFrom shaped glass to thin structures: Fresnel, diffractive, GRIN, metalensesFocusFocus mechanicsFocusing a compound lens: beyond unit focusingFocus actuatorsFocus stacking, macro, and focusing railsAutofocusContrast-detection AFPhase-detection AF (the split-pupil / stereo trick)On-sensor PDAF and dual-pixel AFDepth from focus / defocus, and learned subject AFWhere to focus: saliency, faces, and eyesFocusing in astrophotographyBokeh, focus stacking, and depth-of-field controlRecap: the geometry of focus (pointer, not re-derivation)The bokeh look: shape and structure of the blurExtending DoF: focus stackingControlling and faking DoFFake (synthetic) depth of fieldWhy phones must fake itWhere the depth (or subject) comes fromFrom depth to blur — the thin-lens circleRealistic bokeh — why a Gaussian looks fakeOcclusion-aware compositing and mattingFailure modesGlare suppressionWhere stray light comes from: flare, ghosting, veiling glareHardware suppressionComputational deflare and glare deconvolutionOptical stabilizationThe problem: hand-shake and the blur budgetOptical stabilization: lens-shift vs. sensor-shift (IBIS)Digital / electronic stabilization and the computational alternativesThe eye as an optical instrument: vision and its correctionThe eye as a cameraRefractive errors: the eye out of focusMeasuring the eye's opticsCorrecting visionPresbyopia and the bifocal problem: from bifocals to AF glassesMULTIPLE EXPOSURE IMAGINGDenoising by averagingWhy averaging works: the $1/\sqrt N$ derivationWhen the plain mean is wrong: robust combinationCalibration frames: what averaging can't fixHandheld low light: the phone in your pocketDeep-sky astrophotography: averaging at the extremeHDR mergingThe HDR challengeData capture: how to vary the exposureCurve calibrationCombining exposuresOptimizing the capture and mergeIn-sensor HDR: dynamic range without a bracketApplication to cell phones: HDR+ and burst imagingWhy a phone shoots a burst, and why it underexposesThe HDR+ pipeline: align and robust-merge in rawFrom burst HDR to burst super-resolutionMultiframe or burst super-resolutionThe reconstruction principle: many coarse grids make one fine gridSub-pixel registration is the whole gameFusion: from scattered samples to a sharp imageAccidental versus deliberate offsets: hand tremor and pixel-shift sensorsThe learned eraWhere it runs outManual panorama stitching from multiple viewsThe scenario, and the one rule: rotate, don't translateRefresher: pinhole projection is "divide by depth"Why you don't need 3D: depth cancels for a pure rotationHomographies and homogeneous coordinatesSolving for $H$ from correspondencesWarping and assembling the panoramaAnother application: document flattening and mergingAutomatic panorama stitching from multiple views and feature matchingWhy not brute force, and the two sub-problemsThe feature pipeline, recalled from Part 8RANSAC for a homographyBlendingWhy a hard seam is visible — the photometric mismatchFeathering / alpha blending — and why it ghostsTwo-scale blending — the simple split (the pset method)Multiband / Laplacian-pyramid blending — a transition per bandPoisson / gradient-domain blending — paste gradients, solve for valuesSeam optimization — route the seam instead of fading itThe complete pipeline, end to endBells and whistlesOther projectionsBundle adjustmentMovement and parallax handlingContinuous panoramas (e.g. on cell phones)Incremental registration of a video streamMosaicking a moving strip (and why a central strip)Rolling shutter and exposure driftFocal stacks and depth of field extensionWhy limited depth of field is the problemA simple algorithm: sharpness = local high-frequency energy, then argmaxThe more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)Capturing the stack — hardware, and the magnification trapHyperspectral imaging, color wheelsWhy three numbers aren't enough: RGB as a 3-sample projectionBuilding the spectral stack: filter wheels, tunable filters, pushbroom, snapshotWhat it's for: material ID, agriculture, art and beyondPolarization imagingWhat a camera throws away — polarization as a third axis of lightCapturing the stack — rotate a polarizer, or a polarization mosaicCombining images at different polarizationsIntrinsic images with time lapseThe split, and why one image can't do itWeiss 2001: the median of log-gradientsWhere this chapter belongs: passive vs. active illuminationLucky imaging (planetary / lunar astro)Atmospheric seeing — why one long exposure failsShoot thousands, keep the sharpestAlign and stack the survivorsA poor man's adaptive opticsMANY IMAGES AND PHOTO COLLECTIONSPhoto MosaicsThe tiling-and-matching pipelineColor correction and avoiding repeatsMulti-scale and irregular tilingsWhy it resolves into the target at a distanceRetrievalClassic CBIR — histograms and the text-retrieval analogyDeep retrieval — learned embeddings and CLIPRetrieval at scale — approximate nearest neighborMining what makes a place distinctiveAuto curationTechnical quality — the easy rejectsAesthetics — the hard, learned partGrouping, summary, and diversityArranging the collection, not just culling itLife logging camerasThe devices and how they fireThe memory-prosthesis reframingThe big-data problem passive capture createsPrivacy and ethicsInpainting Using Millions of PhotographsWhy self-similar inpainting isn't enoughScene completion from a huge databaseThe data is the prior — and its modern oppositePhoto tourismStructure-from-motion on internet collectionsFrom reconstruction to experienceLineagePhotobiosAlign and order the collectionLet the data fill the gapsCollection as experienceAverage ExplorerGalton's composite portraiture — and what it was forAlignment is everythingThe average of a categoryAn artistic lineageAverageExplorer: averaging made interactiveWhat an average is, and what it is good forPix 2 GPSGeolocation as retrieval over a geotagged corpusThe answer is a distributionMapping the collection itselfThe learned successorsPersonalized priorsPersonalized restorationPersonalizing generative modelsThe bargain and its ethicsArtistic projects with photo collectionsStatistical collage — SalavonAnticliché cameraPareidoliaDisplaying images togetherSelectionLayoutColor and coherenceThemesVIDEOMotion blur, temporal sampling, and resamplingA frame is an integral over time → motion blurTime is sampled → temporal aliasing, the wagon-wheel effectMotion blur is the temporal prefilter: the two are one tradeoffResampling in time: frame-rate conversionLagrangian vs Eulerian: the organizing distinction for the partVideo compression and motion compensationWhy video compresses far better than still × N: temporal redundancyMotion-compensated prediction: the core trickI, P, and B frames; GOP structureWhy this is "optical flow on a budget"Modern codecs in one breathVideo editingNon-linear editing: the timeline metaphorSummarization: keyframes, fast-forward, and highlightsFun temporal filters: reduce-over-timeTranscript-based editingIn-betweening: synthesizing the frames an edit needsCoda: storyboards, interviews, and where this part landsFrame interpolation and slow-motion synthesisWhy interpolate: faking slow-motion and up-converting frame rateInterpolation = morphing between adjacent framesFlow-based interpolation: warp both frames to the midpoint and blendLearned synthesis: Super SloMo and FILMHybrid low/high resolution and frame-rate imagingThe space–time bandwidth trade-offThe hybrid two-camera architectureMotion from the fast streamApplications: deblur, space–time super-resolution, video from stillsModern descendantsVideo stabilization and rolling-shutter correctionWhat stabilization is: a camera-path signal to be smoothedStage 1: estimating the camera trajectoryStage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)Stage 3: re-rendering and the stabilization↔crop tradeoffRolling-shutter correction: per-row pose and rectificationTime-lapse photographyCapturing a time-lapse: interval, shutter, and day-to-night transitionsDeflickering: stabilizing exposure, white balance, and lightHyperlapse: stabilizing a time-lapse that also movesMining time-lapses from the internetFactoring a time-lapse: reflectance, illumination, and intrinsic imagesVideo texturesFinding good transitionsPlaying itRelatives and descendantsLIGHT FIELDS AND PLENOPTIC CAMERASLight fields 101Capture rays, not pixelsThe plenoptic function, reduced to four dimensionsRay and point are dualReading the 4-D structure through its 2-D slicesRendering a new view by looking up raysLight fields vs. plenoptic and radianceLight field camerasIntegral photography: Lippmann's fly's-eye plate (1908)The plenoptic camera: a microlens array on the sensorLytro: the consumer plenoptic cameraCommercial light-field cameras beyond LytroCamera arrays: a grid of full camerasThe spatial↔angular tradeoffTwo strategies, and why arrays and microlenses are dualOther light field acquisition setupsOne camera on a gantry: sample the aperture in timeHandheld, unstructured capture: let the poses be irregularCatadioptric capture: one sensor, many viewpoints at onceCoded aperture in time: sweep the pupil itselfThe everything-else, mappedRefocusing and synthetic apertureReconstructing a photo: it is all about which rays you sumDigital refocusing is shift-and-addA focal stack from one capture, and an all-in-focus imageFourier-slice photography: the fast versionReading refocus and depth off an epipolar sliceSynthetic aperture: an aperture the size of a roomAberration correction in light fieldsAn aberration is misrouted raysRe-routing each ray to the ideal-lens positionThe trade: ray bookkeeping instead of glass, and only what you sampledLight field aliasing and 4D Fourier analysisThe light field is a sampled signal, and its samples are viewpointsDepth is slope is spectral orientationThe bowtie: a spectrum shaped by the scene's depth rangeHow densely must you sample? The plenoptic-sampling boundGeometry buys back samples: the depth-vs-views tradeoffUnder-sampling looks like a ghostWhere this sitsLumigraph and shape priors for sharper light field renderingPure light-field rendering blurs because it has no shapeThe Lumigraph: reproject onto a geometry proxy, then blendUnstructured inputs: free-hand views, no grid requiredSurface light fieldsLight field microscopyThe optical setup: a microlens array at the intermediate image planeThe spatial-versus-angular trade: a coarse 3-D volumeRecovering the volume: synthetic refocusing, then 3-D deconvolutionWhy single-shot 3-D is the whole pointLight field networksNeRF: a radiance field rendered by volume integrationLight field networks: a ray straight to color, in one evaluationA family of neural light fieldsTest-time training: the network as the per-scene priorHand-off: from rays to radiance fields and generationPractical aspects of light field camerasDo you lose all that resolution?Can you do video? Is it practical? The bandwidth problemThe time dimension is high speedAdjacent frontiers, brieflySo, do I get my camera?MULTI-APERTURE IMAGINGCamera arrays: one rig, many instrumentsOne rig, four instrumentsCommercial arrays: the Light L16Where this sitsBullet timeMulti-camera phonesCOMPUTATIONAL OPTICS AND CODED IMAGINGWavefront codingWhy you cannot just deblur defocusThe fix: re-engineer the blurThe cubic phase plateThe depth-invariant PSF and a single deconvolutionWhat it costsWhere it sitsCompressive sensingSub-Nyquist: fewer measurements than unknownsTwo ingredients: incoherent measurements and sparsityRecovery by $\ell_1$: the geometry of basis pursuitWhy it works: the restricted isometry property, intuitivelyThe single-pixel cameraWhere compressive sensing pays off — and where it does notCoded apertureThe bad forward operator of a clear apertureA mask designed for a flat, zero-free spectrumOne shot, two outputs: depth and an all-in-focus imageThe design criterion: which pattern?Coded-aperture pairs: splitting the trade across two shotsHeterodyning the light field: dappled photographyWhere it sitsPhase-coded aperturesA reminder on wavefront codingFocus sweepThe lattice-focal lensWhere it sitsCode in time (phase, amplitude)Why ordinary motion blur is (almost) unrecoverableThe flutter shutter: chop the exposure into a codeThe decode: one deconvolution, a sharp moving objectAmplitude in time, and phase in timeCoded strobing, temporal multiplexing, and compressive videoMotion-invariant photographyWhere it sitsTheoretical analysis of imaging systems in the 4D light field Fourier domainThe light field's spectrum, and the one move that explains everythingEvery camera is a different slicePutting cameras on one footing: the Bayesian comparisonThe lattice-focal lens: tiling the wedgeThe upper bound — and the gap we have not closedFrom cameras to light transport: the same spectral lensWhere this leaves the partEnd-to-end optimizationThe pipeline as one differentiable graphBackpropagating into the glassWhat gets designed: a height map, not a hyperparameterA gallery of deep-optics resultsWhat it costs, and where it can go wrongWhere it sitsFourier opticsLight as a wave: amplitude, phase, and what "coherent" meansDiffraction is a Fourier transformA lens computes a Fourier transformThe pupil function is the transfer functionThe diffraction limitAberrations are pupil phaseEvery code in this part is a choice of pupilFourier ptychography: synthesizing a bigger pupilExotic / advanced opticsLensless imagingGRINMetalenses and advanced crazy optics à la Barbastathis (coherent though)Non-linear opticsOptical modulators (spatial light modulators): DMD and LCD/LCoSCOMPUTATIONAL SENSORSAssorted pixelsDual-pixel and phase-detect pixels: buying depth and focusClear, white, and other color-filter variantsPolarization pixelsSpatially varying exposure: assorting for dynamic rangeThe common threadModern sensors (quad Bayer, in-sensor HDR, and beyond)Quad Bayer, Tetracell, and nona-binningIn-sensor HDR: capturing range before the mergeDual-pixel autofocus, on the same sensorBSI and stacked sensors: compute under the pixelsThe global-versus-rolling shutter tradeBeyond: nano-prism, organic, and event pixelsOn-sensor HDRStaggered / multiple-exposure readout (DOL-HDR)Dual (and triple) conversion gain (DCG)Split-pixel: a large and a small photodiodeSpatially-varying exposure (SVE) / assorted exposuresLateral overflow integration capacitor (LOFIC)Logarithmic, self-resetting, and counting pixelsOn-sensor HDR versus multi-frame HDRDepth sensorsStereo: passive triangulationStructured light: projecting the textureLiDAR and direct time-of-flightTime of flightPassive depth from one camera, in passingThe menu, in one viewSingle-photon sensors (SPAD, avalanche, photon counting)From avalanche gain to a single-photon clickPhoton-counting arrays: zero read noise, shot-noise-limitedThe Quanta Image Sensor: a different road to one photonWhat it costs: dark counts, dead time, fill factor, data rateDoppler / velocity imagingThe Doppler shift, as a velocity sensorThe instruments: vibrometry, Doppler LiDAR, radarWhere it fitsEvent sensorsHow it works: per-pixel change detectionThe upside: microseconds, dynamic range, no blur, little dataThe downside: no picture, and a new kind of dataUses, lineage, and the contrast with single-photonSpecialized and research sensorsGeiger-mode avalanche-photodiode arrays: photon-counting laser radarDigital-pixel focal-plane arraysScientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaicsStacked and processing-in-pixel sensorsCurved focal planesBeyond the visible, and filter-array sensorsRadiation-hardened and defense focal planesThe chapter's pointExtra sensors and non-visual dataAccelerometer and gyroscope: the inertial measurement unitSound: microphones, audio-visual sync, and the visual microphoneGPS: geotagging and placeCompass and magnetometer: heading and orientationNear-infrared: the cut filter, dark flash, and NIR-assisted denoiseTemperature: dark-current compensationUltra High speed ImagingStreak cameras: sweeping time onto a spatial axisFemto-photography: a movie of light in flightCompressive ultrafast photography: a single-shot coded streakTransient imaging and looking around cornersThe bridge to direct time-of-flight and LiDARCOMPUTATIONAL ILLUMINATIONFlash photographyFlash / no flashRamesh's multiflashRemoving flash artifactsDark flash (plus Stasi version!)High-speed and stroboscopic photographyFreezing motion: the microsecond strobeThe trigger problemStroboscopic multiplicity: a sequence on one frameDigital descendants: LED strobes and high-speed camerasIllumination-based mattingThe well-posed case to beat: chroma keyThe magic prism: Disney's sodium-vapor processNear-infrared and time-multiplexed mattingFlash/no-flash matting: separation by falloffThe throughline: control the capture, not the priorSeparation of Direct and Global IlluminationThe frequency insightNayar's program: programmable, structured illuminationToward coherent separationLight domesThe reflectance field and one-light-at-a-time captureRelighting by linear combinationScaling down: tabletop LED domesScaling out: the dome taken into the wildThe found dome: the eyeAutomatic aesthetic lightingComputational bounce flashDrone lighting: flying the light into placeDual photographyLight transport as a matrixHelmholtz reciprocity: transpose the matrixThe unsettling reach: privacy and seeing the unseenCoherent imagingThe confocal principle: rejecting out-of-focus lightSeeing through scattering mediaThe part in one line3D AND DEPTHMultiple view geometryTwo views: stereo and disparityTwo-view geometry: epipolar lines, and the essential and fundamental matricesWhat "depth" means, and where it comes fromDepth is the z-coordinate, not the ray lengthWhere depth comes from: a cue-and-sensor inventoryRelative vs metric: the scale you usually don't haveMonocular depth estimation (one image → depth)Why one image cannot determine depthRelative depth, and the scale-and-shift ambiguityFrom hand-built priors to borrowed diffusion priorsWhat the maps are good forSingle-image 3-D: tour into the picture, photo pop-up, 3-D Ken BurnsThe universal recipe, and why holes are the hard partTour Into the Picture: the spidery meshAutomatic Photo Pop-up3-D Ken Burns and 3-D photos: the monocular formMulti-view 3-D reconstruction: the classic pipelineThe pipeline, stage by stageWhy it is brittle: and why SfM survives anywayStructured light scanningProjector as inverse camera: triangulation with trivial correspondenceThe coding ladder, and the frames-versus-motion tradeCalibration and failure modesPhotos → radiance fields and Gaussian splatting (NeRF, 3DGS)Two goals, one diagramInverse differentiable rendering, and the discontinuity that forced fuzzinessNeRF: a scene as a tiny neural networkDo we even need the network?3-D Gaussian Splatting: the lessons without the deep learningThe practical recipe: and what is baked inRelaxing the assumptions: NeRF in the wildFeed-forward (amortized) 3-D: skip the per-scene optimizationAmortization: pay once, reuse foreverThe line: DUSt3R, MASt3R, VGGTThe punchline, and the loop it closesThe trade, and the data dependencyRe-photographyWhy you cannot just overlayThe real-time guidance loopWhere it sitsThe landscape, and is 3-D a "fake task"?The field as a landscape, not a lineIs 3-D a "fake task"?The counterpoint, and the frontierINTEGRAL AND IMMERSIVE IMAGINGStereo glassesWheatstone's stereoscope (1838)Routing a different image to each eyeShooting and synthesizing a stereo pairVR gogglesThe three levelsThe optics: a microdisplay and a magnifier per eyeTracking, latency, and why headsets used to make people sickPassthrough, mixed reality, and where today's products sit3D displays with accommodationFour ways to deliver a focus cueThe other gaps, and the ultimate displayLenticular displaysLippmann's integral photography: the common ancestorMulti-view, and the resolution–views tradeoffLight-field telepresence: Google StarlineDisplay depth of field and antialiasingHolographyLippmann and Gabor: recording the waveOff-axis holography and the space-bandwidth wallComputational holographyRetinal projectionThe Maxwellian view: focus set by the display, not the eyeFrom the virtual retinal display to laser eyewearThe frontier: writing to individual conesREVEALING THE INVISIBLEAccidental camerasThe accidental pinhole: a window is a cameraThe accidental pinspeck: the anti-pinholeThe occluder as a crude lens, and recovery as deconvolutionCorners and doorways: an edge that resolves the hidden roomWhere else the world hides a cameraReflections in the eyeThe cornea as a catadioptric mirrorThe geometry: from a corneal pixel to a direction in the worldWhat the recovered reflection is good forEyes for relightingMotion and video magnificationThe Lagrangian precursor: track, then exaggerateEulerian video magnification: amplify the time series at each pixelWhy amplifying brightness amplifies motionPhase-based magnification: move the motion into phaseWhat it reveals: vital signs, structures, materials, modesVisual microphoneFrom sub-pixel motion to a sound waveformBandwidth: high-speed cameras and the rolling-shutter trickHow good is the copy? The object's frequency responseThe active cousins: laser vibrometry and interferometryCorner cameraThe edge as a one-dimensional apertureFrom a faint gradient to a usable signalWhat the corner can and cannot tell youActive non-line-of-sightThird-bounce geometry and time-of-flightThe hardware: photographing light in flightFrom back-projection to fast, exact inversionThe trade, stated plainlyPassive non-line-of-sightThe occluder is what makes it solvableA deconvolution where the lens is unknownOne dimension: the corner cameraTwo dimensions from a single photo: computational periscopyActive versus passive, the ledgerMm-wave, wifiWhy radio walks through wallsTime of flight, again: radar is NLOS with a longer waveFrom a radio smear to a human skeleton: the learned mapWhat it sees, and what it costs usADJACENT FIELDS AND APPLICATIONSOptical computingAstroExtreme long exposureTracking, stacking, and selection (pointers)X-rayMedicalMicroscopyMm-waveMusic, soundFluorescenceOpto-acousticUltrasoundAerial imagingComputer visionRobotics, drivingHUMAN FACTORSHuman factors and the art of photographyMake better photosTypical shooting scenariosMacro photographySpecial effect photographyFun artsy stuffPerception of artEthics of computational photographyComputational models of perceptionSpatial (and spatio-temporal) visionUser studiesAccessibility: photography by and for blind usersBlind camera — capture without a sighted operatorThe social and personal practice of photographyIMAGE FORENSICS AND AUTHENTICATIONImage ForensicsThe problem and the threat modelSensor and pipeline traces: PRNU, CFA, and noiseCompression, geometry, and metadata forensicsDeepfakes, GAN/diffusion fingerprints, and learned detectionWhy forensics is evidence, not proof — and the hand-off to provenanceAuthentication and Provenance (C2PA)From detection to attestation: trustworthy cameras and watermarkingC2PA and Content Credentials: the standardAI disclosure, watermarking, and the regulatory pushLimits, critiques, and the forensics partnershipSYSTEMSProgrammable and modular camerasImage processing librariesLightroom-style raw developersPhotoshop-style editorsNetworking and image transportPhotography programming on phonesPERFORMANCE ENGINEERING AND HALIDE8-bit and fixed-point arithmeticInteger versus fixed-point: where the binary point sitsRounding, dithering, and the banding trapSaturation and the width of the accumulatorfp16 versus bf16: the exponent–mantissa bargainint8 quantization for neural inferenceWhen float is non-negotiableAlgorithmic speedupsSeparability: a 2-D pass for the price of two 1-D passesRecursive / IIR filters: a running state, cost independent of radiusIntegral images / summed-area tables: any box sum in four lookupsFast median filters: a sliding histogram for a constant-time medianPyramids and multiscale: do the large-scale work on small imagesDiscretize the range to accelerate non-linear filtersDownsampling and edge-aware upsamplingWhere this goes nextAutomatic search for fast methodsThe problem is a curve, not a pointWhy search beats a hand-shrunk CNNMa et al. 2022: searching structure and parameters togetherThree axes of "search instead of design"The broader family, and what it costsWhere this goes nextModern CPUs: memory hierarchy, parallelism, and what it takes to go fastWhy moving data, not doing math, is the bottleneckThe memory hierarchy and localityThe roofline: is my kernel compute- or memory-bound?The forms of parallelismWhat it takes to leverage a modern machineWhy photography is hard for the machineWhere this goes nextHardware backends: GPU, NPU, DSPGPU: the data-parallel workhorseThe neural accelerator: NPU and TPUDSP: the real-time control loopISP, FPGA, and ASIC: the fixed-function endThe heterogeneous SoC, and the spectrum to carry awayOn-device ML runtimes, and why the work stays on the phoneHalide: Decoupling Algorithms from SchedulesWhat it means to separate the algorithm from the scheduleWhy image pipelines are uniquely hard to optimizeThe split was always there: done by handThe scheduling spaceAuto-scheduling: letting the compiler searchResults, impact, and reachGradient Halide: differentiating the pipelineWhere this goes nextHalide programmingThe three nouns: Func, Var, ExprA first image pipeline: brighten, then blurReductions: RDom, sums, and histogramsThe default schedule, and seeing the loopsScheduling the loops within a stage: reorder, split, tile, vectorize, unroll, parallel, fuseThe heart of it: producer–consumer granularity (compute_at, store_at)The schedule ladder, with numbers: ten times faster from one lineBoundaries and boundsSame algorithm, new machine: the GPULetting the compiler schedule: the auto-schedulerThe development loop: correctness, measurement, and benchmarking hygieneEfficient neural network inferenceQuantization: fewer bits per weightPruning: fewer weightsKnowledge distillation: a small student, a big teacherLow-rank and tensor factorizationEfficient architectures: cheapness designed inNeural architecture search: let the machine design itHardware-aware deployment and the rooflineWhere this leaves the partCONCLUSIONS, DISCUSSIONRecap in contextModern phones, multiple apertures, pano, HDR+Recap: a modern mirrorless cameraRecap: a modern cell phone multi cameraLightroomPhotoshopWhy phones are so good (at photography)Computation beats glassThe whole pipeline is co-designedMachine learning and data at scaleMany small cameras for one big oneThe human and system advantagesThe hard caveats: physics still wins where it mustThe throughlineBACK MATTERBibliographyGlossaryAcronymsTerm indexAPPENDICESRefreshersLinear algebraCalculus: derivatives, gradients, integralsOptimization and regressionProbability and information theoryMachine learning and deep learningProgramming: Python, C++, and PyTorchProblem Set 0 — Environment and C++ basicsSummaryInstallation and Environment SetupC++SubmissionProblem Set 1 — Image class, point operations, and colorSummaryThe Image ClassBrightness and ContrastMore Image Class MethodsColorspacesSpanish Castle IllusionWhite BalanceProblem Set 2 — Convolution and the bilateral filterSummarySmart AccessorBlurringDenoising using Bilateral FilteringExtra creditSubmissionProblem Set 3 — Denoising and demosaickingSummaryDenoising from a sequence of imagesDemosaicingEdge-based greenRed and blue based on green6.865 only (or 5% Extra Credit): Sergey Prokudin-GorskyExtra credit (maximum of 10%)Problem Set 4 — High dynamic rangeSummaryHDR mergingTone mappingExtra credit (10% max)Problem Set 5 — Resampling, warping, and morphingSummaryResamplingWarping and morphingExtra creditProblem Set 6 — Homographies and manual panoramasSummaryClass MorphHomogeneous CoordinatesLinear AlgebraWarp and Image with a HomographyCompute Homography from 4 Pairs of PointsBounding boxesExtra Credit (up to 10% total)Problem Set 7 — Automatic panoramasSummaryPrevious Problem Set CodeClass MorphHarris Corner DetectionDescriptor and correspondencesRANSACAutomatic panorama stitchingBlendingMini planet6.8370: Stitch N Images (6.8371: Extra Credit 5%)Make your own panoramaExtra credits (10% max)Problem Set 8 — Non-photorealistic renderingSummaryPaintbrush splattingPainterly renderingOriented painterly renderingYour imagePaper Review (6.865 only)Extra creditsProblem Set 9 — Make-your-own, video, and ethicsSummaryMake Your Own AssignmentEthical issues in computational photographyAssignment ListsEXIF and image metadataWhat EXIF isThe fields, grouped by what they describeHow far to trust itPrivacy: the metadata that follows the pictureReading and writing EXIFDNG: the Digital NegativeWhat DNG is, and the problem it solvesInside the containerWhat the raw payload looks like: mosaic vs linearThe color recipe: matrices, profiles, and white balanceOpcodes: corrections the decoder must applyCompression, and embedding the originalWhere you meet DNG: adoption and relativesTrade-offs, and DNG's relation to EXIFRendering a raw DNGThe pipeline, step by stepWhere Lightroom's "look" actually comes fromWhat we would ask an Adobe engineer to checkTwo kinds of DNG, and the special casesEasy mistakes (most of which we made)DatasetsClassification and featuresSuper-resolutionDeblurring and restorationDenoisingHDR and tone mappingRetouching and enhancementDepth and motionLight fieldsColor and white balanceFacesInpainting, segmentation, and mattingImage qualityA camera-feature wish listExposure, ISO, and dynamic rangeBracketing more than exposureFocus and depth of fieldComputational raw and the sensorMotion data, metadata, and workflowPanorama and multi-shotThe interface and the ecosystemHow this book was createdTwo documents, not oneCompiling a sectionFigures as codeGenerative imagery: cover art and 3DVerification and reviewKeeping a long book coherentThe toolchainWhat the machine did, and what it did notWho wrote what: a per-part estimateThe course tutor: a local, book-grounded AI teaching assistantWhat it is, and what it is forLocal-firstGrounded in the book: retrieval-augmented generationIt links, it shows equations, it shows figuresTwo front-ends, one coreWhat the instructor seesPrivacy and candorThe semi-automatic grading systemThe shape (to be confirmed)Automatic versus human (to be confirmed)To be filled in (from the instructor)Under the hood: prompts, patterns, and verifiersPrompt patterns that made it workThe verifier suiteWhy this is the interesting partReading a Lytro light fieldThe container: the LFP/LFR formatThe pipeline, step by stepFrom hexagonal lenslets to a uniform gridWhy a white imageOpen questions and where we approximateEasy mistakes (most of which we made)File conversion toolsJPG → PNG, without an alpha channelDNG → a linear PNGA Lytro capture → a Stanford-style light-field archiveHow they run in the browserEasy mistakesThe interactive figures — a prompt-by-prompt making-ofSummaryLorentz resonance — fig-lorentz-resonance (Figure 2.1.8)Rainbow droplet — fig-rainbow-droplet (Figure 2.1.14)Diffraction wave simulation — fig-diffraction-wave-sim (Figure 2.1.29)Exposure-triangle simulator — fig-exposure-triangle-sim (Figure 2.11.5)Repeated quantization — fig-repeated-quantization (Figure 3.1.4)JPEG generation loss — fig-jpeg-generation-loss (Figure 3.17.9)Exposure round-trip (JPEG) — fig-exposure-jpeg-roundtrip (Figure 3.17.10)1-D sampling pipeline — fig-sampling-1d-demo (Figure 3.10.4)2-D sampling pipeline — fig-sampling-2d-pipeline (Figure 3.10.6)Mitchell–Netravali bicubic (B,C) — fig-bicubic-bc (Figure 3.11.13)Rotation resample challenge — fig-rotation-resample-interactive (Figure 3.11.18)Mini-Lightroom — fig-mini-lightroom (Figure 3.18.4)Poisson blending — fig-poisson-blend (Figure 5.1.5)Bilateral grid (3-D) — fig-bilateral-grid-3d (Figure 5.2.15)Perspective montage — fig-perspective-montage (Figure 6.6.13)Beier–Neely morph — fig-beier-neely-demo (Figure 6.3.4)Live face landmarks — fig-face-landmarks-live (Figure 7.10.2)Lens optimizer — fig-lens-optimizer-demo (Figure 9.4.4)Full auto-panorama — fig-pano-stitch (Figure 10.7.9)CLIP-IQA curation — fig-clip-iqa (Figure 11.3.2)Photobio time-lapse — fig-photobio-demo (Figure 11.7.2)Refocus shift geometry — fig-refocus-shift-geometry (Figure 13.4.2)Chroma key — fig-chroma-key (Figure 8.11.7)Portrait-lighting simulator — fig-portrait-lighting-sim (Figure 2.11.26)Aberration explorer — fig-aberration-explorer (Figure 9.1.6)Interactive demo indexSharing, linking, and embedding a demoIntroductionFundamentals — light, optics, sensors, colorBasic image processing and the ISPComputational tools — machine learning and diffusionEdges matter — gradient-domain and edge-preservingWarping and morphingMatching pixels across space and timeSingle-image computational photographyOptics, lenses, and aberration correctionMultiple-exposure imaging — HDR and panoramasMany images and photo collectionsVideoLight fields and plenoptic cameras3-D and depthAppendices and end matterNot yet placedBACK MATTER
Glossary Acronyms Index 💬 Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.×
Computational Photography, an AI-powered Slopendium
« Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte. »
“If I had more time, I would have written you a shorter letter.”
Blaise Pascal
31 parts · 290 chapters · 254 drafted · blue = drafted, links open the chapter; grey = outline only
This is a very preliminary draft, and even the organization is still in flux. Feedback is welcome on content and organization, but fine-grained critique (wording, typography, line-level polish) is probably premature.
To leave feedback, select any text on a page and click the note / highlight button that pops up, or open the annotation panel with the tab at the top-right (‹). Notes are visible only inside our private review group.
Preliminary draft · v0.1.240
show Parts Chapters Sections collapse all open in Chapter pages Section pages
1.1 From Digital to Computational Photography 1.1.1 Computation is the new optics 1.1.2 The goals of computational photography 1.1.3 What makes a technique useful 1.1.4 A field in transition: from hand designed algorithms to learning to generative AI 1.1.5 Does photography still matter in the age of generative AI? 1.1.6 A field at a crossroads 1.1.7 It's not just about photography 1.2 How to read this book 1.2.1 Machine-readable by design 1.3 How this book was made 1.4 What programming language for computational photography? 1.4.1 On the desktop: Python, C++, and Halide 1.4.2 In the browser: JavaScript and the web stack 1.4.3 On the phone: Android and iOS 1.4.4 Libraries 1.4.5 Vibe coding 1.4.6 Programmability, or the lack thereof 1.5 Problem sets 2.1 Light and optics 2.1.1 Light: rays, waves, and the spectrum 2.1.2 How light is created 2.1.3 Light power and brightness 2.1.4 Summing coherent vs. incoherent light 2.1.5 Reflection, refraction, and what happens at a surface 2.1.6 Light-matter interaction is largely resonance 2.1.7 Polarization 2.1.8 The color of objects: illumination times reflectance 2.1.9 The BRDF and the look of materials 2.1.10 Illuminants 2.1.11 Radiometry: radiance, irradiance, exposure, and falloff 2.1.12 Global illumination 2.1.13 Wave effects, diffraction, and the diffraction limit 2.2 Pinhole image formation and linear perspective 2.2.1 Pinhole imaging and the perspective projection 2.2.2 Homogeneous coordinates 2.2.3 Camera in a general configuration 2.2.4 Intrinsics, extrinsics, and what cropping really does 2.2.5 What perspective preserves, and what it destroys 2.2.6 Wide-angle distortion: spheres bulge and faces stretch at the edges 2.2.7 Photography with focal length: framing, magnification, and compression 2.2.8 Depth, ray length, and unprojection 2.3 Lens image formation 2.3.1 Single lenses: refraction and Snell's law 2.3.2 Thin lens optics 2.3.3 Aperture and the f-number 2.4 Image measurements as integrals 2.4.1 The plenoptic function 2.4.2 The pixel integral 2.4.3 Exposure values (EV) and stops 2.4.4 Splitting the integral 2.5 Depth of field 2.5.1 The circle of confusion 2.5.2 Depth of field versus depth of focus 2.5.3 Choosing the maximum circle of confusion 2.5.4 The double cone, and the near and far limits 2.5.5 Hyperfocal distance 2.5.6 The surprising invariance 2.5.7 Background blur beyond the focus plane 2.5.8 Defocus is not a blur of the image 2.5.9 Sensor size scales depth of field 2.6 Motion blur 2.6.1 Blur from subject motion 2.6.2 Blur from camera shake 2.6.3 How shift-invariant is camera-shake blur? 2.6.4 Which dominates: translation or rotation? 2.6.5 The hand-holding rule of thumb 2.6.6 Shake from the shutter and mirror 2.6.7 Panning: fighting subject motion by moving the camera 2.6.8 Freezing motion with light 2.7 Sensors: photosites, CCD vs CMOS 2.7.1 The photoelectric effect 2.7.2 Photon to number: the photosite 2.7.3 Beyond the visible spectrum: near-infrared, thermal, and ultraviolet 2.7.4 Analog-to-digital conversion 2.7.5 CCD versus CMOS 2.7.6 Time integration: the exposure 2.7.7 Shutters 2.7.8 A taste of modern sensor tricks 2.7.9 The impact of sensor size 2.7.10 How far sensors have come 2.8 Noise, signal-to-noise ratio and dynamic range 2.8.1 Noise sources 2.8.2 The algebra of noise 2.8.3 Signal-to-noise ratio: it's about ratios 2.8.4 Noise variance is affine in pixel brightness 2.8.5 Noise as a function of ISO 2.8.6 Dynamic range 2.8.7 Regimes, and how close we are to the limits 2.9 Imaging as a linear system 2.9.1 Linear systems from a billboard to an image 2.9.2 Linear systems from ray space (light fields) to an image 2.9.3 Space–time: imaging across time 2.9.4 When linearity breaks down 2.9.5 Where this is going: invertibility 2.10 Imaging as an inverse problem 2.10.1 Linear algebra: how hard is this problem? 2.10.2 Priors and the manifold of natural images 2.10.3 Probability: the Bayesian view 2.10.4 Optimization, inference, loss 2.10.5 Deep learning, amortized inference, and generative AI 2.10.6 When you can design the operator — and a teaser of information theory 2.10.7 Recap: what makes imaging hard, and where it's going 2.10.8 Harder inverse problems: factorization 3.1 Anatomy of the eye and image formation 3.2 Perceptual color and trichromatic vision 3.2.1 Cone Photoreceptors and Perceptual Color 3.2.2 Metamers 3.2.3 Color blindness 3.2.4 Opponent process and the multistage model 3.2.5 So what are the primary colors? 3.2.6 Color as linear algebra 3.3 Measuring and encoding color 3.3.1 analysis vs synthesis and non-orthogonality 3.3.2 Measuring color 3.3.3 Chromaticity diagram 3.3.4 Linear vs Gamma vs. log encoding 3.3.5 RGB color spaces 3.3.6 Non-linear perceptually-uniform color space 3.3.7 HSV, HSL, and cylindrical color spaces 3.3.8 Reproducing color 3.3.9 Skin tones 3.4 Sensing color: multiplexing strategies 3.4.1 Multiplexing in time 3.4.2 Multiplexing in space: the color filter array 3.4.3 Multiplexing across separate sensors: the beam-splitter 3.4.4 Multiplexing per pixel by dispersion: color routers and nano-prisms 3.4.5 Multiplexing per pixel by tunable absorbers: quantum dots 3.4.6 Multiplexing in depth: stacked photodiodes 3.4.7 Hybrid strategies 3.4.8 Beyond trichromatic capture: full spectrum and multispectral 3.4.9 Spectral sensitivity: matching the eye, catching photons, and white balance 3.5 Visual processing 3.5.1 Light adaptation 3.5.2 Lightness constancy 3.5.3 Color constancy 3.5.4 Contrast 3.5.5 Spatial vision 3.5.6 Temporal vision 3.5.7 Attention and eye movements 3.5.8 The visual system makes things up 3.6 Animal eyes 3.6.1 The optics: many ways to form an image 3.6.2 The pupil: aperture shape as a clue to ecology 3.6.3 Amphibious vision: focusing in air and in water 3.6.4 The sensor: a retina that is not uniform 3.6.5 Seeing in the dark: sensitivity and its tricks 3.6.6 Acuity: the finest detail, and how to sharpen it 3.6.7 Color vision: opsins remixed 3.6.8 Polarization: the axis we are blind to 3.6.9 Temporal vision: flicker, and motion first 4.1 Exposure 4.1.1 Basic photography: exposure settings — shutter, aperture, and ISO 4.1.2 Metering: from printed guides to the sensor 4.1.3 f-stops versus T-stops 4.1.4 From exposure to photometry 4.1.5 Exposure modes, UI, and auto-ISO 4.2 Lenses 4.2.1 Lenses and focal length 4.2.2 Aperture and the diaphragm 4.2.3 Focusing the lens 4.2.4 Image stabilization 4.2.5 Keeping the lens clean 4.2.6 Lens filters: polarizers, ND, and graduated ND 4.3 Focus, autofocus, and depth of field 4.3.1 Manual-focus aids: making the focal plane visible 4.4 Cameras 4.4.1 UI and display 4.4.2 How the viewfinder works 4.4.3 Anatomy of a modern full-frame interchangeable-lens camera 4.4.4 Anatomy of cell-phone cameras 4.4.5 Camera versus phone 4.4.6 Types of cameras 4.4.7 Cameras beyond photography 4.4.8 A camera's other sensors 4.5 Video 4.5.1 Cinema lenses 4.6 Illumination and the flash 4.6.1 Natural illumination 4.6.2 Flash 4.6.3 Flash metering 4.6.4 Indirect illumination and bounce flash 4.6.5 Multiple-point lighting 4.7 Traditional and Digital Darkroom 4.7.1 The traditional darkroom 4.7.2 The digital darkroom 4.8 Limitations of the medium 4.8.1 Compensation, accentuation, conflict 4.8.2 The picture is flat: depth and its cues 4.8.3 The picture is static: time and motion 4.8.4 One viewpoint, and a finite frame 4.8.5 The contrast is limited: dynamic range and gamut 4.8.6 Resolution: the limitation we are overcoming 4.9 Displays 4.9.1 The range of displays 4.9.2 Display technology 4.9.3 HDR displays and dual modulation 4.9.4 The film movie projector 4.9.5 Printing 4.9.6 Why the display characteristics matter 4.9.7 Distance, resolution and acuity 4.9.8 Sharpening for the display: size and distance 4.9.9 Light level viewing conditions 4.9.10 Robustness of perspective to the viewer's viewpoint 4.9.11 Gamut and gamut mapping 4.9.12 Color management, ICC, and industry standards 4.10 Photographs are usually not passive objective recordings 4.10.1 Faithful is not the same as realistic 4.10.2 Choosing the instant, choosing the viewpoint 4.10.3 Speed, aperture, and focus: the photographer rules 4.10.4 Filters and lighting: building the scene's light 4.10.5 The darkroom: tone, dodging, and burning 4.10.6 Make-up and retouching: authoring the subject 4.10.7 Getting closer — systematic capture, and the biases that remain 4.11 Types of photography 4.11.1 People 4.11.2 Places 4.11.3 Animals 4.11.4 Motion and the decisive instant 4.11.5 Objects and the arranged frame 4.11.6 Low light and the sky 4.11.7 Art and abstraction 4.11.8 Technique and apparatus 4.11.9 One frame, or a body of work 4.12 Photography and videography jobs 4.12.1 The market: what the image is for 4.12.2 The role: where in the pipeline 4.12.3 The employment model 4.12.4 A job is a triple 4.12.5 How the opportunities have shifted 4.12.6 The crew, collapsing toward the individual 4.13 Why people take photos 4.13.1 A small map of motives 4.13.2 From a memory technology to a communication one 4.13.3 What the psychology actually shows 4.13.4 When people take photos, and why the usual numbers are not evidence 4.13.5 Why this belongs in a book about how images are made 5.1 Image representation 5.1.1 An image is an array 5.1.2 Float vs 8-bit vs more bits 5.1.3 Stochastic Quantization and Dithering 5.1.4 How the array sits in memory 5.1.5 Pixel coordinate conventions 5.1.6 Alpha and extra channels 5.1.7 Video and more dimensions 5.1.8 Pixel lookups and boundaries 5.1.9 Beyond the pixels: basic metadata and EXIF 5.1.10 Three kinds of operation 5.2 What the numbers mean 5.2.1 What the numbers actually mean: encoding 5.2.2 What the numbers actually mean: color spaces 5.3 Developing, Testing and Debugging 5.3.1 The workflow: build, run, look at the picture 5.3.2 Principles 5.3.3 Test on inputs you can verify by hand 5.3.4 Finding good real test inputs 5.3.5 Per-algorithm debug recipes 5.3.6 Crashes and bounds 5.3.7 Vibe coding: writing image code with an LLM 5.4 Point operations 5.4.1 Three kinds of operation 5.4.2 The point-operation curve 5.4.3 Exposure: a multiply in linear light 5.4.4 Brightness vs. exposure 5.4.5 Contrast: steepen about a pivot 5.4.6 Black point, white point, and levels 5.4.7 The general tone curve 5.4.8 Basic color enhancement: saturation and vibrance 5.4.9 Converting to black and white 5.4.10 Lookup tables 5.4.11 Video color grading 5.4.12 Local point operations 5.5 Histograms 5.5.1 the histogram 5.5.2 the histogram depends on the encoding space 5.5.3 a histogram is a sampled estimate 5.5.4 histogram equalization 5.5.5 histogram matching 5.5.6 constraining the slope: Ward's histogram-based tone mapping 5.6 Global tone mapping 5.6.1 The dynamic-range problem 5.6.2 Two families: global and local 5.6.3 Global tone mapping: one curve for everything 5.6.4 Histogram equalization and Ward's bound 5.6.5 Local tone mapping and the halo problem: why naive local fails 5.6.6 Why log space pays off 5.6.7 The analog ancestor: the Zone System 5.6.8 HDR on every screen: the gain map 5.6.9 Where this is going 5.7 Beauty curves: the camera "look" 5.7.1 Why scene-linear looks dull 5.7.2 The shape of the curve 5.7.3 Highlights: roll off to white, not to a color 5.7.4 Saturation, and the rest of the "look" 5.7.5 The discipline 5.8 Neighborhood operations and convolution 5.8.1 Motivation: blur and sharpen 5.8.2 Convolution 101 5.8.3 The flip: where-from vs. where-to 5.8.4 Properties: the impulse, normalization, symmetry, commutativity 5.8.5 Nitty-gritty: finite images 5.8.6 A blur zoo 5.8.7 Separability 5.8.8 Gradient and oriented filters 5.8.9 Is imaging blur usually a shift-invariant convolution? 5.8.10 Where this goes next 5.9 Sharpening 5.9.1 Linear sharpening 5.9.2 Why it's called "unsharp mask" 5.9.3 Sharpen in gamma, linear, or log? 5.9.4 Non-linear sharpening 5.10 Fourier 5.10.1 Images as vectors in a high-dimensional space 5.10.2 why Fourier 5.10.3 Definition: one coefficient per wave 5.10.4 Sines are the eigenvectors of convolution 5.10.5 The 2-pixel example 5.10.6 Reading an image's Fourier transform 5.10.7 A small bestiary of transforms 5.10.8 The fine print: two limitations of Fourier 5.10.9 Windowing: the price of measuring a real spectrum 5.10.10 Application preview: can we deblur, given the blur? 5.11 Sampling, downsampling, and aliasing 5.11.1 Aliasing: a high frequency in disguise 5.11.2 Nyquist and the sampling theorem 5.11.3 In two dimensions, aliasing has a direction 5.11.4 Sampling in the frequency domain: spectral replicas 5.11.5 Seeing it: sampling and pre-filtering, hands on 5.11.6 The ideal reconstruction filter: sinc, and why it is unreachable 5.12 Resampling and upsampling 5.12.1 Domain operations: moving pixels around 5.12.2 Start with scaling up 5.12.3 The naive idea, and why it leaves gaps 5.12.4 Loop over the output, use the inverse transform 5.12.5 Nearest neighbor 5.12.6 Linear interpolation, in 1-D first 5.12.7 Bilinear, in 2-D 5.12.8 The convolution perspective 5.12.9 Better kernels: bicubic and Lanczos 5.12.10 Upsampling vs. downsampling: scale the kernel 5.12.11 Prefiltering when the transform isn't a clean scale 5.12.12 Beyond Nyquist-limited reconstruction 5.12.13 Where this goes next 5.13 Pyramids and wavelets 5.13.1 Halfway between space and frequency 5.13.2 The idea: process an image at many resolutions 5.13.3 The Gaussian pyramid 5.13.4 The Laplacian pyramid 5.13.5 Reconstruction: an exact encoder and decoder 5.13.6 Each level is a frequency band 5.13.7 What the bands look like on a real image 5.13.8 Candid limitations 5.13.9 Cousins: wavelets and steerable pyramids 5.13.10 Pyramid blending 5.13.11 Other applications 5.13.12 Multiresolution as a recurring theme 5.14 Image metrics 5.14.1 Full-reference vs. no-reference 5.14.2 why not just L2? 5.14.3 PSNR 5.14.4 SSIM 5.14.5 VDP and HDR-VDP 5.14.6 learned metrics 5.14.7 The right metric depends on the task 5.15 Denoising basics 5.15.1 what is noise? 5.15.2 denoising by averaging multiple shots 5.15.3 denoising from a single image 5.15.4 Spatial averaging and its limits 5.15.5 The bilateral filter: averaging by affinity 5.15.6 Denoising in the pyramid: coring 5.15.7 Denoise color more than brightness 5.15.8 noise estimation 5.15.9 The limits of denoising 5.16 Demosaicking 5.16.1 Reminder: the Bayer mosaic 5.16.2 The task: full RGB at every pixel 5.16.3 The naive approach: interpolate each channel on its own 5.16.4 Why naive interpolation zippers: averaging across an edge 5.16.5 Doing better: edge-directed interpolation 5.16.6 The harder half: red and blue, and color fringing 5.16.7 Green-based demosaicking: interpolate the color difference 5.16.8 Classic (non-learning) demosaicking: the general strategy 5.16.9 Related: the optical anti-aliasing filter 5.16.10 Quad-Bayer sensors: remosaic before demosaicking 5.16.11 Fuji X-Trans: a larger, irregular CFA 5.16.12 Beyond hand-tuned: joint denoising and learned demosaicking 5.16.13 Cross-reference: other ways to sense color 5.16.14 Where this sits in the pipeline 5.17 Auto-exposure and auto white balance 5.17.1 Auto-exposure: metering 5.17.2 white balance and color constancy 5.17.3 Automatic white balance 5.17.4 The limits of white balance, and CRI 5.18 File formats and compression 5.18.1 The big picture: none, lossless, lossy 5.18.2 Data versus metadata: EXIF 5.18.3 PNG: the format we read from 5.18.4 JPEG: compression by perception 5.18.5 RAW files: before the cooking 5.18.6 HDR formats: more than 8 bits 5.18.7 Modern formats 5.18.8 Other formats: TIFF and GIF 5.19 Recap ISP, non-destructive editing: 5.19.1 A basic ISP 5.19.2 Pipeline design and tuning 5.19.3 The ISP, evolving: traditional → learned → generative 5.19.4 Recap 2: non-destructive editing 6 COMPUTATIONAL TOOLS
6.1 Linear Inverse Problems and Regression6.1.1 Blur is linear, so deblurring is inversion6.1.2 Images as vectors, and the notation overload6.1.3 Regression: deblurring as least squares6.1.4 Matrices without forming matrices: gradient descent and conjugate gradient6.1.5 Efficient solvers6.2 Fundamental limits of inverse problems6.2.1 The question: what can no algorithm recover?6.2.2 The SVD picture: null space, near-null, and the noise explosion6.2.3 Information bounds: a floor no estimator beats6.2.4 The resolution–noise tradeoff, and how a prior invents6.2.5 Fundamental versus practical, and coded capture as the escape6.3 Information theory6.3.1 The basics, at a high level6.3.2 Compression: the clearest application6.3.3 The imaging system as a channel6.3.4 Information-theoretic quantities as objectives6.3.5 The modern revival: learned compression and generative models6.3.6 Cousins, not children6.3.7 Where it is least relevant: aesthetics and perception6.4 Machine learning6.4.1 The framing: learned operators replace hand-designed ones6.4.2 The data story: synthetic data, noise models, datasets6.5 Deep learning6.5.1 Low-level learned operators (pixel-to-pixel)6.5.2 Mid- and high-level learned predictors6.5.3 Generative models: image-to-image translation6.5.4 Learned perceptual metrics and losses6.6 Generative AI and diffusion6.6.1 The framing: generation is learning and sampling a prior $p(x)$6.6.2 Diffusion: generation as iterated denoising6.6.3 Conditioning: text, images, and control6.6.4 Posterior sampling: generative priors for inverse problems6.6.5 Other generative families, in brief6.6.6 Caveats and ethics6.7 Image priors6.7.1 A catalog of priors: from smoothness to learned6.7.2 Shift and scale invariance6.7.3 Anisotropy: sparsity in a single direction6.7.4 Linear color models6.7.5 Denoising as a universal prior: Plug-and-Play and RED6.7.6 Diffusion is iterated denoising (the continuous limit)6.7.7 Score-based models, the thin manifold, and imperfect priors7 EDGES MATTER
7.1 Bilateral filtering7.1.1 Motivation: local tone-mapping haloes7.1.2 Bilateral filter7.1.3 Cross / joint bilateral7.1.4 Bilateral grid7.1.5 Bilateral-grid learning (HDRnet)7.2 Non-local means7.3 Edge-preserving optimization — colorization7.3.1 The shift: filtering becomes optimization7.3.2 Colorization: the canonical demonstration7.3.3 Writing down the energy7.3.4 Solving it: a sparse linear system we already own7.3.5 One affinity, two faces: the matting Laplacian connection7.4 Local Laplacian filters7.4.1 The wish, and why the bilateral leaves a halo7.4.2 What a Laplacian pyramid hands you7.4.3 The mechanism: a remapping recomputed per output pixel7.4.4 One curve, three edits: enhancement, tone mapping, inverse tone mapping7.4.5 Contrast with the bilateral base/detail split7.4.6 The cost, and the fast version7.5 Matting Laplacian7.5.1 The matting problem: compositing, and why it is under-determined7.5.2 The color-line assumption: α becomes linear in color7.5.3 Closed-form matting: the matting Laplacian7.5.4 Spectral matting: eigenvectors of the matting Laplacian7.5.5 An alternative matting Laplacian7.5.6 Beyond matting: spatially-varying white balance7.6 Guided image filtering7.6.1 The key idea: the output is a local line in the guide7.6.2 Fitting the line, and the one knob7.6.3 Why it is fast: O(N) regardless of window size7.6.4 No gradient reversal7.6.5 Reading the affinity back out7.6.6 Where it is used7.7 Locally adaptive regression kernel (LARK)7.7.1 Change the verb: from averaging to fitting7.7.2 Steering the kernel to the local structure7.7.3 The regression flavor of the affinity7.7.4 LARK as a structure descriptor7.8 Poisson image editing7.8.1 Edit gradients, not pixels7.8.2 Seamless cloning: the headline application7.8.3 Reconstruction = solving the Poisson equation7.8.4 Poisson vs. pyramid blending7.8.5 Advanced techniques7.9 Seam optimization7.9.1 Find the non-edges: least-noticeable cuts7.9.2 Intelligent scissors / live-wire: the cut as a least-cost path7.9.3 Snakes (active contours): the continuous cousin7.9.4 Graph cut: globally optimal boundaries by min-cut / max-flow7.9.5 GrabCut: interactive cutout from a single rectangle7.9.6 Normalized cuts: the spectral relaxation7.9.7 Seam carving: content-aware resize by DP7.9.8 Time-lapse via DP: a seam through time7.9.9 Video textures: looping a seam in time7.9.10 GraphCut textures & photomontage: cut, then blend7.9.11 Where this connects: MRFs and beyond7.10 Recap: which edge-aware technique when?7.10.1 The three relationships to edges7.10.2 Pros, cons, and cost: when to reach for each7.10.3 A worked chooser8 WARPING, MORPHING, AND LAGRANGIAN APPROACHES
8.1 Warping8.1.1 Warping is a domain transform: move *where*, not *what*8.1.2 Forward vs inverse warping: why output-driven wins8.1.3 Resampling: a quick reminder (developed in BASIC)8.1.4 Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladder8.1.5 Specifying the warp II: free-form warps from sparse correspondences8.1.6 Liquify: the warp with a paintbrush bolted on8.2 Antialiasing for complex transforms8.2.1 When the footprint stops being a square: the warp Jacobian8.2.2 EWA: the elliptical weighted average8.2.3 Feline and anisotropic MIP probing: EWA on a GPU budget8.2.4 Other footprint integrators: summed-area tables and ripmaps8.2.5 Footprint estimation, clamping, and when to reach for it8.3 Morphing8.3.1 Why a cross-dissolve isn't enough: the ghosting motivation8.3.2 Two interpolations: domain (shape) and range (color)8.3.3 The morphing recipe (combine both)8.3.4 Field morphing: the Beier–Neely line-pair warp8.3.5 Mesh / triangulation morphing (the alternative warp)8.3.6 View morphing: the geometrically-correct in-between of two views8.3.7 Recap and significance8.4 Morphable models8.4.1 Step 1: dense correspondence, the precondition for averaging shapes8.4.2 Step 2: Procrustes alignment, factor out pose so PCA sees shape8.4.3 Step 3: PCA, a mean and a basis of eigen-deformations8.4.4 Step 4: shape and appearance are *separate* bases8.4.5 Step 5: fitting a new photo, analysis-by-synthesis8.4.6 Step 6: once fitted, edit by moving in the space8.4.7 Step 7: from linear PCA to neural priors8.4.8 Recap and significance8.5 Shape-preserving warping8.5.1 Why plain interpolation shears: the rigidity gap8.5.2 As-rigid-as-possible (ARAP) manipulation8.5.3 Linear blend skinning and why the weights decide everything8.5.4 Bounded biharmonic weights (BBW)8.5.5 ARAP interpolation: rigid-as-possible morphing between two poses8.5.6 Moving least squares: the simpler cousin8.5.7 Recap and significance8.6 Perspective distortion and its correction8.6.1 Keystoning is projection, not a lens flaw8.6.2 The fix is a homography: re-render the façade fronto-parallel8.6.3 The optical alternative at capture: tilt-shift / Scheimpflug8.6.4 Cropping wide-angle photographs: recenter with a homography, don't just crop8.6.5 A different perspective distortion: wide-angle portraits, and a content-aware fix8.6.6 Beyond faces: correcting arbitrary objects (MaDCoW)8.6.7 A montage has no single viewpoint: a family in a box8.6.8 Recomposing perspective after capture: computational zoom8.6.9 The catch: resampling cost, and "only a plane rectifies exactly"8.6.10 Where this sits: one map, then transport9 MATCHING PIXELS AND HUMANS ACROSS SPACE AND TIME
9.1 Brute force9.1.1 Why you must align first9.1.2 Brute-force translational alignment (SSD / NCC)9.1.3 Phase correlation: the whole shift from one FFT9.1.4 Coarse-to-fine alignment on a pyramid9.2 Sub-pixel matching9.2.1 Refining the cost: fit a parabola9.2.2 Refining in the frequency domain: phase correlation9.2.3 Refining by gradient: the Lucas–Kanade step9.2.4 What breaks it: peak-locking, texture, and noise9.2.5 One tool, everywhere downstream9.3 Sparse matching9.3.1 Where to look: corners, and the structure tensor9.3.2 Invariance: surviving scale and rotation9.3.3 Describing a neighborhood: SIFT and its zoo9.3.4 Matching: nearest neighbor in descriptor space9.4 Feature tracking9.4.1 Tracking vs dense flow: sparse-but-long vs dense-but-short9.4.2 KLT = Lucas–Kanade, per feature, iterated over time9.4.3 Good features to track = where the structure tensor is well-conditioned9.4.4 What breaks long-term tracking: drift, appearance change, occlusion, and re-detection9.4.5 Modern point trackers (briefly)9.4.6 An application: synthetic motion blur from a track9.5 Robustness: the ratio test and RANSAC9.5.1 The ratio test: reject ambiguous matches before fitting9.5.2 RANSAC: fit from a minimal random sample, score by consensus9.5.3 Variants and degeneracy9.6 Deep learning approaches to sparse matching9.6.1 Learned local features: SIFT's job, done by a network9.6.2 Learned matching — reason about the whole set at once9.6.3 The 3-D-aware turn: pointmaps subsume matching9.7 Fast matching9.7.1 Approximate nearest neighbors for sparse descriptors9.7.2 PatchMatch: randomized dense correspondence by propagation9.7.3 Fast high-dimensional matching by random projection9.8 Optical flow9.8.1 What optical flow is: and whether it is even well-defined9.8.2 Brightness constancy and the optical-flow constraint9.8.3 The aperture problem9.8.4 Lucas–Kanade: local constant-flow least squares (and the structure tensor)9.8.5 Horn–Schunck: global smoothness regularization9.8.6 Large motion: coarse-to-fine warping9.8.7 Learned flow: RAFT (neuralize the classical pipeline)9.9 Deep learning approaches to optical flow9.9.1 The unrolling principle: neuralize the classical solver9.9.2 Cost volumes and warping inside the net9.9.3 RAFT and the recurrent update9.10 Face tracking9.10.1 Detecting the face9.10.2 Landmarks: pinning down the features9.10.3 Tracking across time9.10.4 Lifting to 3-D: the morphable model9.10.5 Recognition, and the dark side9.10.6 Which library to use9.11 Body pose estimation9.11.1 Top-down vs bottom-up9.11.2 On-device, real time9.11.3 Lifting to 3-D: parametric bodies9.11.4 Which library to use10 SINGLE IMAGE COMPUTATIONAL PHOTOGRAPHY
10.1 Denoising10.1.1 A reminder: bilateral filtering and wavelet/pyramid shrinkage10.1.2 BM3D: group similar patches and filter them together10.1.3 Learning to denoise10.2 Super-resolution10.2.1 What problem super-resolution solves (and why it's ill-posed)10.2.2 Scenarios: single-image, burst, and hybrid space–time10.2.3 Reconstruction vs hallucination: measured detail vs invented detail10.3 Demosaicking and joint reconstruction10.3.1 The pipeline problem10.3.2 FlexISP: one energy, one prior10.3.3 Learning the joint prior: the founding formulation10.3.4 Model-based and unrolled networks10.3.5 Learning where the hard cases are10.3.6 Co-designing the mosaic: a learnable sensor10.3.7 The end-to-end learned ISP10.3.8 Better backbones: CNN → transformer → diffusion prior10.3.9 Exotic mosaics: learned remosaicking10.3.10 Where do the labels come from?10.3.11 Correcting chromatic aberration jointly10.3.12 Recap10.4 Non-blind deblurring10.4.1 Deblurring in the presence of noise: why naive inversion fails10.4.2 The Wiener filter — the regularized, noise-aware inverse10.4.3 Sparse gradients: the prior that keeps edges sharp10.5 Blind deblurring10.5.1 Blind deblurring: estimating the kernel *and* the image10.5.2 A more realistic blur model: spatially-varying (camera-shake) blur10.5.3 Engineering the aperture: depth and all-focus from a coded mask10.6 Dehazing10.6.1 Dehazing as a prior-driven inverse problem10.6.2 Differentiable image pipelines and algorithm optimization (Halide)10.7 Mixed-lighting white balance10.7.1 Why a global gain must fail10.7.2 Estimating the per-pixel mixture: an under-determined inverse problem10.7.3 Correcting each region for its own light10.7.4 What came after10.8 Inpainting, texture synthesis, and object removal10.8.1 Inpainting as filling unmeasured pixels — the spectrum of priors10.8.2 PDE / diffusion-based inpainting10.8.3 Texture synthesis (Efros–Leung; Efros–Freeman quilting)10.8.4 Exemplar inpainting: clone, healing brush, and object removal (Criminisi)10.8.5 Data-driven scene completion (Hays & Efros)10.8.6 Deep inpainting (context encoders → partial/gated conv → diffusion)10.8.7 Highlight / specular recovery10.8.8 Epitomes — a compact patch model10.9 Patch match10.9.1 The nearest-neighbor field, and why exhaustive search is the bottleneck10.9.2 PatchMatch — randomized correspondence [@barnes-etal-2009|Barnes et al. 2009]10.9.3 Applications: hole filling, retargeting, reshuffle10.9.4 Shift-Map image editing [@pritch-etal-2009|Pritch et al. 2009] — editing as graph-cut labeling10.10 Colorization10.10.1 One channel in, three channels out10.10.2 A spectrum of priors: scribbles, references, and learned models10.10.3 The multimodality trap: why naïve colorization goes muddy10.10.4 Closing the loop: learned priors with a human's hints10.10.5 Plausible is not correct10.10.6 Where it sits10.11 Compositing, segmentation and matting10.11.1 Compositing and alpha channels10.11.2 Segmentation: cutting the object out10.11.3 The fundamental matting equation10.11.4 Blue/green-screen matting and chroma keying10.11.5 Traditional matting approaches10.11.6 Deep-learning matting10.11.7 Generative matting10.11.8 Harmonization and where the blends live10.11.9 Optical effects beyond alpha10.12 Illumination related effects in a single image10.12.1 Intrinsic images: the unifying frame ($I = R\cdot S$)10.12.2 Multiple-light / spatially-varying white balance10.12.3 Reflection removal — pulling apart a transmission and a glass reflection10.12.4 Shadow detection and shadow removal10.12.5 Specular-highlight removal / "fake polarization" (the dichromatic model)10.13 Tone Mapping10.13.1 Global vs local, re-hashed10.13.2 A smarter global curve: histogram adjustment10.13.3 A taxonomy of local methods10.13.4 The darkroom ancestor: dodge & burn and the Zone System10.14 Style transfer10.14.1 Classical style transfer: patches and statistics10.14.2 Neural style and feed-forward stylization10.14.3 Style transfer as image-to-image translation10.15 Non-photorealistic rendering10.15.1 What NPR is for, and the one idea10.15.2 Stroke-based / painterly rendering (and the brush p-set)10.15.3 Edge-preserving abstraction: bilateral + Difference-of-Gaussians10.15.4 Example-based stylization and the bridge to neural style10.15.5 Region-based stylization: stained glass, low-poly, mosaics10.15.6 Artistic screening and halftoning11 COMPOUND LENSES, AND ABERRATION CORRECTION
11.1 Aberrations and optical challenges11.1.1 Taxonomy of challenges11.1.2 Spherical aberration11.1.3 Coma11.1.4 Astigmatism11.1.5 Field curvature11.1.6 Chromatic aberration11.1.7 Radial distortion11.1.8 Wave effects and diffraction11.1.9 Vignetting11.1.10 Flare and coating11.2 Aberrations correction11.2.1 The two families of cure11.2.2 Correction in glass11.2.3 Computational correction11.2.4 Radial distortion correction11.3 Measuring lens quality11.3.1 The Modulation Transfer Function (MTF)11.3.2 Measuring MTF in practice11.3.3 Spot diagrams and the PSF11.3.4 MTF through focus and depth of focus11.3.5 Field-dependent MTF: sagittal vs tangential11.3.6 Color, geometry, and illumination measurements11.3.7 Scalar quality summaries11.4 Lens optimization11.4.1 The high-level idea: design as optimization11.4.2 The forward model: ray-tracing and spot diagrams11.4.3 From hand calculation to software11.4.4 Tradeoffs: the design is always a compromise11.4.5 Tolerancing: from the nominal design to a manufacturable one11.5 A short bestiary of classic designs11.5.1 The lens as a system: cardinal points, pupils, f-number, T-stop11.6 Scaling laws in optics11.6.1 Lohmann's scaling laws: why a good lens is heavy11.6.2 The gigapixel barrier for a single lens11.6.3 The escape: monocentric multiscale optics11.6.4 Capture everything, crop later: the spatial cousin of the light field11.7 Special optics11.7.1 Tilt-shift and the Scheimpflug principle11.7.2 Fisheye and non-rectilinear projection11.7.3 Mirrors: catadioptric and reflecting systems11.7.4 Periscope / folded-lens design (smartphone telephoto)11.7.5 Anamorphic optics11.7.6 Stereo (3D) lenses11.7.7 Macro, microscope objectives, and telescopes11.7.8 Teleconverters11.7.9 From shaped glass to thin structures: Fresnel, diffractive, GRIN, metalenses11.8 Focus11.8.1 Focus mechanics11.8.2 Focusing a compound lens: beyond unit focusing11.8.3 Focus actuators11.8.4 Focus stacking, macro, and focusing rails11.9 Autofocus11.9.1 Contrast-detection AF11.9.2 Phase-detection AF (the split-pupil / stereo trick)11.9.3 On-sensor PDAF and dual-pixel AF11.9.4 Depth from focus / defocus, and learned subject AF11.9.5 Where to focus: saliency, faces, and eyes11.9.6 Focusing in astrophotography11.10 Bokeh, focus stacking, and depth-of-field control11.10.1 Recap: the geometry of focus (pointer, not re-derivation)11.10.2 The bokeh look: shape and structure of the blur11.10.3 Extending DoF: focus stacking11.10.4 Controlling and faking DoF11.11 Fake (synthetic) depth of field11.11.1 Why phones must fake it11.11.2 Where the depth (or subject) comes from11.11.3 From depth to blur — the thin-lens circle11.11.4 Realistic bokeh — why a Gaussian looks fake11.11.5 Occlusion-aware compositing and matting11.11.6 Failure modes11.12 Glare suppression11.12.1 Where stray light comes from: flare, ghosting, veiling glare11.12.2 Hardware suppression11.12.3 Computational deflare and glare deconvolution11.13 Optical stabilization11.13.1 The problem: hand-shake and the blur budget11.13.2 Optical stabilization: lens-shift vs. sensor-shift (IBIS)11.13.3 Digital / electronic stabilization and the computational alternatives11.14 The eye as an optical instrument: vision and its correction11.14.1 The eye as a camera11.14.2 Refractive errors: the eye out of focus11.14.3 Measuring the eye's optics11.14.4 Correcting vision11.14.5 Presbyopia and the bifocal problem: from bifocals to AF glasses12 MULTIPLE EXPOSURE IMAGING
12.1 Denoising by averaging12.1.1 Why averaging works: the $1/\sqrt N$ derivation12.1.2 When the plain mean is wrong: robust combination12.1.3 Calibration frames: what averaging can't fix12.1.4 Handheld low light: the phone in your pocket12.1.5 Deep-sky astrophotography: averaging at the extreme12.2 HDR merging12.2.1 The HDR challenge12.2.2 Data capture: how to vary the exposure12.2.3 Curve calibration12.2.4 Combining exposures12.2.5 Optimizing the capture and merge12.2.6 In-sensor HDR: dynamic range without a bracket12.3 Application to cell phones: HDR+ and burst imaging12.3.1 Why a phone shoots a burst, and why it underexposes12.3.2 The HDR+ pipeline: align and robust-merge in raw12.3.3 From burst HDR to burst super-resolution12.4 Multiframe or burst super-resolution12.4.1 The reconstruction principle: many coarse grids make one fine grid12.4.2 Sub-pixel registration is the whole game12.4.3 Fusion: from scattered samples to a sharp image12.4.4 Accidental versus deliberate offsets: hand tremor and pixel-shift sensors12.4.5 The learned era12.4.6 Where it runs out12.5 Manual panorama stitching from multiple views12.5.1 The scenario, and the one rule: rotate, don't translate12.5.2 Refresher: pinhole projection is "divide by depth"12.5.3 Why you don't need 3D: depth cancels for a pure rotation12.5.4 Homographies and homogeneous coordinates12.5.5 Solving for $H$ from correspondences12.5.6 Warping and assembling the panorama12.5.7 Another application: document flattening and merging12.6 Automatic panorama stitching from multiple views and feature matching12.6.1 Why not brute force, and the two sub-problems12.6.2 The feature pipeline, recalled from Part 812.6.3 RANSAC for a homography12.7 Blending12.7.1 Why a hard seam is visible — the photometric mismatch12.7.2 Feathering / alpha blending — and why it ghosts12.7.3 Two-scale blending — the simple split (the pset method)12.7.4 Multiband / Laplacian-pyramid blending — a transition per band12.7.5 Poisson / gradient-domain blending — paste gradients, solve for values12.7.6 Seam optimization — route the seam instead of fading it12.7.7 The complete pipeline, end to end12.8 Bells and whistles12.8.1 Other projections12.8.2 Bundle adjustment12.8.3 Movement and parallax handling12.9 Continuous panoramas (e.g. on cell phones)12.9.1 Incremental registration of a video stream12.9.2 Mosaicking a moving strip (and why a *central* strip)12.9.3 Rolling shutter and exposure drift12.10 Focal stacks and depth of field extension12.10.1 Why limited depth of field is the problem12.10.2 A simple algorithm: sharpness = local high-frequency energy, then argmax12.10.3 The more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)12.10.4 Capturing the stack — hardware, and the magnification trap12.11 Hyperspectral imaging, color wheels12.11.1 Why three numbers aren't enough: RGB as a 3-sample projection12.11.2 Building the spectral stack: filter wheels, tunable filters, pushbroom, snapshot12.11.3 What it's for: material ID, agriculture, art and beyond12.12 Polarization imaging12.12.1 What a camera throws away — polarization as a third axis of light12.12.2 Capturing the stack — rotate a polarizer, or a polarization mosaic12.12.3 Combining images at different polarizations12.13 Intrinsic images with time lapse12.13.1 The split, and why one image can't do it12.13.2 Weiss 2001: the median of log-gradients12.13.3 Where this chapter belongs: passive vs. active illumination12.14 Lucky imaging (planetary / lunar astro)12.14.1 Atmospheric seeing — why one long exposure fails12.14.2 Shoot thousands, keep the sharpest12.14.3 Align and stack the survivors12.14.4 A poor man's adaptive optics13 MANY IMAGES AND PHOTO COLLECTIONS
13.1 Photo Mosaics13.1.1 The tiling-and-matching pipeline13.1.2 Color correction and avoiding repeats13.1.3 Multi-scale and irregular tilings13.1.4 Why it resolves into the target at a distance13.2 Retrieval13.2.1 Classic CBIR — histograms and the text-retrieval analogy13.2.2 Deep retrieval — learned embeddings and CLIP13.2.3 Retrieval at scale — approximate nearest neighbor13.2.4 Mining what makes a place distinctive13.3 Auto curation13.3.1 Technical quality — the easy rejects13.3.2 Aesthetics — the hard, learned part13.3.3 Grouping, summary, and diversity13.3.4 Arranging the collection, not just culling it13.4 Life logging cameras13.4.1 The devices and how they fire13.4.2 The memory-prosthesis reframing13.4.3 The big-data problem passive capture creates13.4.4 Privacy and ethics13.5 Inpainting Using Millions of Photographs13.5.1 Why self-similar inpainting isn't enough13.5.2 Scene completion from a huge database13.5.3 The data is the prior — and its modern opposite13.6 Photo tourism13.6.1 Structure-from-motion on internet collections13.6.2 From reconstruction to experience13.6.3 Lineage13.7 Photobios13.7.1 Align and order the collection13.7.2 Let the data fill the gaps13.7.3 Collection as experience13.8 Average Explorer13.8.1 Galton's composite portraiture — and what it was for13.8.2 Alignment is everything13.8.3 The average of a category13.8.4 An artistic lineage13.8.5 AverageExplorer: averaging made interactive13.8.6 What an average is, and what it is good for13.9 Pix 2 GPS13.9.1 Geolocation as retrieval over a geotagged corpus13.9.2 The answer is a distribution13.9.3 Mapping the collection itself13.9.4 The learned successors13.10 Personalized priors13.10.1 Personalized restoration13.10.2 Personalizing generative models13.10.3 The bargain and its ethics13.11 Artistic projects with photo collections13.11.1 Statistical collage — Salavon13.11.2 Anticliché camera13.12 Pareidolia13.13 Displaying images together13.13.1 Selection13.13.2 Layout13.13.3 Color and coherence13.13.4 Themes14 VIDEO
14.1 Motion blur, temporal sampling, and resampling14.1.1 A frame is an integral over time → motion blur14.1.2 Time is sampled → temporal aliasing, the wagon-wheel effect14.1.3 Motion blur *is* the temporal prefilter: the two are one tradeoff14.1.4 Resampling in time: frame-rate conversion14.1.5 Lagrangian vs Eulerian: the organizing distinction for the part14.2 Video compression and motion compensation14.2.1 Why video compresses far better than still × N: temporal redundancy14.2.2 Motion-compensated prediction: the core trick14.2.3 I, P, and B frames; GOP structure14.2.4 Why this is "optical flow on a budget"14.2.5 Modern codecs in one breath14.3 Video editing14.3.1 Non-linear editing: the timeline metaphor14.3.2 Summarization: keyframes, fast-forward, and highlights14.3.3 Fun temporal filters: reduce-over-time14.3.4 Transcript-based editing14.3.5 In-betweening: synthesizing the frames an edit needs14.3.6 Coda: storyboards, interviews, and where this part lands14.4 Frame interpolation and slow-motion synthesis14.4.1 Why interpolate: faking slow-motion and up-converting frame rate14.4.2 Interpolation = morphing between adjacent frames14.4.3 Flow-based interpolation: warp both frames to the midpoint and blend14.4.4 Learned synthesis: Super SloMo and FILM14.5 Hybrid low/high resolution and frame-rate imaging14.5.1 The space–time bandwidth trade-off14.5.2 The hybrid two-camera architecture14.5.3 Motion from the fast stream14.5.4 Applications: deblur, space–time super-resolution, video from stills14.5.5 Modern descendants14.6 Video stabilization and rolling-shutter correction14.6.1 What stabilization is: a camera-path signal to be smoothed14.6.2 Stage 1: estimating the camera trajectory14.6.3 Stage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)14.6.4 Stage 3: re-rendering and the stabilization↔crop tradeoff14.6.5 Rolling-shutter correction: per-row pose and rectification14.7 Time-lapse photography14.7.1 Capturing a time-lapse: interval, shutter, and day-to-night transitions14.7.2 Deflickering: stabilizing exposure, white balance, and light14.7.3 Hyperlapse: stabilizing a time-lapse that also moves14.7.4 Mining time-lapses from the internet14.7.5 Factoring a time-lapse: reflectance, illumination, and intrinsic images14.8 Video textures14.8.1 Finding good transitions14.8.2 Playing it14.8.3 Relatives and descendants15 LIGHT FIELDS AND PLENOPTIC CAMERAS
15.1 Light fields 10115.1.1 Capture rays, not pixels15.1.2 The plenoptic function, reduced to four dimensions15.1.3 Ray and point are dual15.1.4 Reading the 4-D structure through its 2-D slices15.1.5 Rendering a new view by looking up rays15.1.6 Light fields vs. plenoptic and radiance15.2 Light field cameras15.2.1 Integral photography: Lippmann's fly's-eye plate (1908)15.2.2 The plenoptic camera: a microlens array on the sensor15.2.3 Lytro: the consumer plenoptic camera15.2.4 Commercial light-field cameras beyond Lytro15.2.5 Camera arrays: a grid of full cameras15.2.6 The spatial↔angular tradeoff15.2.7 Two strategies, and why arrays and microlenses are dual15.3 Other light field acquisition setups15.3.1 One camera on a gantry: sample the aperture in time15.3.2 Handheld, unstructured capture: let the poses be irregular15.3.3 Catadioptric capture: one sensor, many viewpoints at once15.3.4 Coded aperture in time: sweep the pupil itself15.3.5 The everything-else, mapped15.4 Refocusing and synthetic aperture15.4.1 Reconstructing a photo: it is all about which rays you sum15.4.2 Digital refocusing is shift-and-add15.4.3 A focal stack from one capture, and an all-in-focus image15.4.4 Fourier-slice photography: the fast version15.4.5 Reading refocus and depth off an epipolar slice15.4.6 Synthetic aperture: an aperture the size of a room15.5 Aberration correction in light fields15.5.1 An aberration is misrouted rays15.5.2 Re-routing each ray to the ideal-lens position15.5.3 The trade: ray bookkeeping instead of glass, and only what you sampled15.6 Light field aliasing and 4D Fourier analysis15.6.1 The light field is a sampled signal, and its samples are viewpoints15.6.2 Depth is slope is spectral orientation15.6.3 The bowtie: a spectrum shaped by the scene's depth range15.6.4 How densely must you sample? The plenoptic-sampling bound15.6.5 Geometry buys back samples: the depth-vs-views tradeoff15.6.6 Under-sampling looks like a ghost15.6.7 Where this sits15.7 Lumigraph and shape priors for sharper light field rendering15.7.1 Pure light-field rendering blurs because it has no shape15.7.2 The Lumigraph: reproject onto a geometry proxy, then blend15.7.3 Unstructured inputs: free-hand views, no grid required15.7.4 Surface light fields15.8 Light field microscopy15.8.1 The optical setup: a microlens array at the intermediate image plane15.8.2 The spatial-versus-angular trade: a coarse 3-D volume15.8.3 Recovering the volume: synthetic refocusing, then 3-D deconvolution15.8.4 Why single-shot 3-D is the whole point15.9 Light field networks15.9.1 NeRF: a radiance field rendered by volume integration15.9.2 Light field networks: a ray straight to color, in one evaluation15.9.3 A family of neural light fields15.9.4 Test-time training: the network as the per-scene prior15.9.5 Hand-off: from rays to radiance fields and generation15.10 Practical aspects of light field cameras15.10.1 Do you lose all that resolution?15.10.2 Can you do video? Is it practical? The bandwidth problem15.10.3 The time dimension is high speed15.10.4 Adjacent frontiers, briefly15.10.5 So, do I get my camera?16 MULTI-APERTURE IMAGING
16.1 Camera arrays: one rig, many instruments16.1.1 One rig, four instruments16.1.2 Commercial arrays: the Light L1616.1.3 Where this sits16.2 Bullet time16.3 Multi-camera phones17 COMPUTATIONAL OPTICS AND CODED IMAGING
17.1 Wavefront coding17.1.1 Why you cannot just deblur defocus17.1.2 The fix: re-engineer the blur17.1.3 The cubic phase plate17.1.4 The depth-invariant PSF and a single deconvolution17.1.5 What it costs17.1.6 Where it sits17.2 Compressive sensing17.2.1 Sub-Nyquist: fewer measurements than unknowns17.2.2 Two ingredients: incoherent measurements and sparsity17.2.3 Recovery by $\ell_1$: the geometry of basis pursuit17.2.4 Why it works: the restricted isometry property, intuitively17.2.5 The single-pixel camera17.2.6 Where compressive sensing pays off — and where it does not17.3 Coded aperture17.3.1 The bad forward operator of a clear aperture17.3.2 A mask designed for a flat, zero-free spectrum17.3.3 One shot, two outputs: depth and an all-in-focus image17.3.4 The design criterion: which pattern?17.3.5 Coded-aperture pairs: splitting the trade across two shots17.3.6 Heterodyning the light field: dappled photography17.3.7 Where it sits17.4 Phase-coded apertures17.4.1 A reminder on wavefront coding17.4.2 Focus sweep17.4.3 The lattice-focal lens17.4.4 Where it sits17.5 Code in time (phase, amplitude)17.5.1 Why ordinary motion blur is (almost) unrecoverable17.5.2 The flutter shutter: chop the exposure into a code17.5.3 The decode: one deconvolution, a sharp moving object17.5.4 Amplitude in time, and phase in time17.5.5 Coded strobing, temporal multiplexing, and compressive video17.5.6 Motion-invariant photography17.5.7 Where it sits17.6 Theoretical analysis of imaging systems in the 4D light field Fourier domain17.6.1 The light field's spectrum, and the one move that explains everything17.6.2 Every camera is a different slice17.6.3 Putting cameras on one footing: the Bayesian comparison17.6.4 The lattice-focal lens: tiling the wedge17.6.5 The upper bound — and the gap we have not closed17.6.6 From cameras to light transport: the same spectral lens17.6.7 Where this leaves the part17.7 End-to-end optimization17.7.1 The pipeline as one differentiable graph17.7.2 Backpropagating into the glass17.7.3 What gets designed: a height map, not a hyperparameter17.7.4 A gallery of deep-optics results17.7.5 What it costs, and where it can go wrong17.7.6 Where it sits17.8 Fourier optics17.8.1 Light as a wave: amplitude, phase, and what "coherent" means17.8.2 Diffraction is a Fourier transform17.8.3 A lens computes a Fourier transform17.8.4 The pupil function *is* the transfer function17.8.5 The diffraction limit17.8.6 Aberrations are pupil phase17.8.7 Every code in this part is a choice of pupil17.8.8 Fourier ptychography: synthesizing a bigger pupil17.9 Exotic / advanced opticsoutline 17.9.1 Lensless imaging17.9.2 GRIN17.9.3 Metalenses and advanced crazy optics à la Barbastathis (coherent though)17.9.4 Non-linear optics17.10 Optical modulators (spatial light modulators): DMD and LCD/LCoSoutline 18 COMPUTATIONAL SENSORS
18.1 Assorted pixels18.1.1 Dual-pixel and phase-detect pixels: buying depth and focus18.1.2 Clear, white, and other color-filter variants18.1.3 Polarization pixels18.1.4 Spatially varying exposure: assorting for dynamic range18.1.5 The common thread18.2 Modern sensors (quad Bayer, in-sensor HDR, and beyond)18.2.1 Quad Bayer, Tetracell, and nona-binning18.2.2 In-sensor HDR: capturing range before the merge18.2.3 Dual-pixel autofocus, on the same sensor18.2.4 BSI and stacked sensors: compute under the pixels18.2.5 The global-versus-rolling shutter trade18.2.6 Beyond: nano-prism, organic, and event pixels18.3 On-sensor HDR18.3.1 Staggered / multiple-exposure readout (DOL-HDR)18.3.2 Dual (and triple) conversion gain (DCG)18.3.3 Split-pixel: a large and a small photodiode18.3.4 Spatially-varying exposure (SVE) / assorted exposures18.3.5 Lateral overflow integration capacitor (LOFIC)18.3.6 Logarithmic, self-resetting, and counting pixels18.3.7 On-sensor HDR versus multi-frame HDR18.4 Depth sensors18.4.1 Stereo: passive triangulation18.4.2 Structured light: projecting the texture18.4.3 LiDAR and direct time-of-flight18.4.4 Time of flight18.4.5 Passive depth from one camera, in passing18.4.6 The menu, in one view18.5 Single-photon sensors (SPAD, avalanche, photon counting)18.5.1 From avalanche gain to a single-photon click18.5.2 Photon-counting arrays: zero read noise, shot-noise-limited18.5.3 The Quanta Image Sensor: a different road to one photon18.5.4 What it costs: dark counts, dead time, fill factor, data rate18.6 Doppler / velocity imaging18.6.1 The Doppler shift, as a velocity sensor18.6.2 The instruments: vibrometry, Doppler LiDAR, radar18.6.3 Where it fits18.7 Event sensors18.7.1 How it works: per-pixel change detection18.7.2 The upside: microseconds, dynamic range, no blur, little data18.7.3 The downside: no picture, and a new kind of data18.7.4 Uses, lineage, and the contrast with single-photon18.8 Specialized and research sensors18.8.1 Geiger-mode avalanche-photodiode arrays: photon-counting laser radar18.8.2 Digital-pixel focal-plane arrays18.8.3 Scientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaics18.8.4 Stacked and processing-in-pixel sensors18.8.5 Curved focal planes18.8.6 Beyond the visible, and filter-array sensors18.8.7 Radiation-hardened and defense focal planes18.8.8 The chapter's point18.9 Extra sensors and non-visual data18.9.1 Accelerometer and gyroscope: the inertial measurement unit18.9.2 Sound: microphones, audio-visual sync, and the visual microphone18.9.3 GPS: geotagging and place18.9.4 Compass and magnetometer: heading and orientation18.9.5 Near-infrared: the cut filter, dark flash, and NIR-assisted denoise18.9.6 Temperature: dark-current compensation18.10 Ultra High speed Imaging18.10.1 Streak cameras: sweeping time onto a spatial axis18.10.2 Femto-photography: a movie of light in flight18.10.3 Compressive ultrafast photography: a single-shot coded streak18.10.4 Transient imaging and looking around corners18.10.5 The bridge to direct time-of-flight and LiDAR19 COMPUTATIONAL ILLUMINATION
19.1 Flash photography19.1.1 Flash / no flash19.1.2 Ramesh's multiflash19.1.3 Removing flash artifacts19.1.4 Dark flash (plus Stasi version!)19.2 High-speed and stroboscopic photography19.2.1 Freezing motion: the microsecond strobe19.2.2 The trigger problem19.2.3 Stroboscopic multiplicity: a sequence on one frame19.2.4 Digital descendants: LED strobes and high-speed cameras19.3 Illumination-based matting19.3.1 The well-posed case to beat: chroma key19.3.2 The magic prism: Disney's sodium-vapor process19.3.3 Near-infrared and time-multiplexed matting19.3.4 Flash/no-flash matting: separation by falloff19.3.5 The throughline: control the capture, not the prior19.4 Separation of Direct and Global Illumination19.4.1 The frequency insight19.4.2 Nayar's program: programmable, structured illumination19.4.3 Toward coherent separation19.5 Light domes19.5.1 The reflectance field and one-light-at-a-time capture19.5.2 Relighting by linear combination19.5.3 Scaling down: tabletop LED domes19.5.4 Scaling out: the dome taken into the wild19.5.5 The found dome: the eye19.6 Automatic aesthetic lighting19.6.1 Computational bounce flash19.6.2 Drone lighting: flying the light into place19.7 Dual photography19.7.1 Light transport as a matrix19.7.2 Helmholtz reciprocity: transpose the matrix19.7.3 The unsettling reach: privacy and seeing the unseen19.8 Coherent imaging19.8.1 The confocal principle: rejecting out-of-focus light19.8.2 Seeing through scattering media19.8.3 The part in one line20 3D AND DEPTH
20.1 Multiple view geometry20.1.1 Two views: stereo and disparity20.1.2 Two-view geometry: epipolar lines, and the essential and fundamental matrices20.2 What "depth" means, and where it comes from20.2.1 Depth is the *z*-coordinate, not the ray length20.2.2 Where depth comes from: a cue-and-sensor inventory20.2.3 Relative vs metric: the scale you usually don't have20.3 Monocular depth estimation (one image → depth)20.3.1 Why one image cannot determine depth20.3.2 Relative depth, and the scale-and-shift ambiguity20.3.3 From hand-built priors to borrowed diffusion priors20.3.4 What the maps are good for20.4 Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns20.4.1 The universal recipe, and why holes are the hard part20.4.2 Tour Into the Picture: the spidery mesh20.4.3 Automatic Photo Pop-up20.4.4 3-D Ken Burns and 3-D photos: the monocular form20.5 Multi-view 3-D reconstruction: the classic pipeline20.5.1 The pipeline, stage by stage20.5.2 Why it is brittle: and why SfM survives anyway20.6 Structured light scanning20.6.1 Projector as inverse camera: triangulation with trivial correspondence20.6.2 The coding ladder, and the frames-versus-motion trade20.6.3 Calibration and failure modes20.7 Photos → radiance fields and Gaussian splatting (NeRF, 3DGS)20.7.1 Two goals, one diagram20.7.2 Inverse differentiable rendering, and the discontinuity that forced fuzziness20.7.3 NeRF: a scene as a tiny neural network20.7.4 Do we even need the network?20.7.5 3-D Gaussian Splatting: the lessons without the deep learning20.7.6 The practical recipe: and what is baked in20.7.7 Relaxing the assumptions: NeRF in the wild20.8 Feed-forward (amortized) 3-D: skip the per-scene optimization20.8.1 Amortization: pay once, reuse forever20.8.2 The line: DUSt3R, MASt3R, VGGT20.8.3 The punchline, and the loop it closes20.8.4 The trade, and the data dependency20.9 Re-photography20.9.1 Why you cannot just overlay20.9.2 The real-time guidance loop20.9.3 Where it sits20.10 The landscape, and is 3-D a "fake task"?20.10.1 The field as a landscape, not a line20.10.2 Is 3-D a "fake task"?20.10.3 The counterpoint, and the frontier21 INTEGRAL AND IMMERSIVE IMAGING
21.1 Stereo glasses21.1.1 Wheatstone's stereoscope (1838)21.1.2 Routing a different image to each eye21.1.3 Shooting and synthesizing a stereo pair21.2 VR goggles21.2.1 The three levels21.2.2 The optics: a microdisplay and a magnifier per eye21.2.3 Tracking, latency, and why headsets used to make people sick21.2.4 Passthrough, mixed reality, and where today's products sit21.3 3D displays with accommodation21.3.1 Four ways to deliver a focus cue21.3.2 The other gaps, and the ultimate display21.4 Lenticular displays21.4.1 Lippmann's integral photography: the common ancestor21.4.2 Multi-view, and the resolution–views tradeoff21.4.3 Light-field telepresence: Google Starline21.4.4 Display depth of field and antialiasing21.5 Holography21.5.1 Lippmann and Gabor: recording the wave21.5.2 Off-axis holography and the space-bandwidth wall21.5.3 Computational holography21.6 Retinal projection21.6.1 The Maxwellian view: focus set by the display, not the eye21.6.2 From the virtual retinal display to laser eyewear21.6.3 The frontier: writing to individual cones22 REVEALING THE INVISIBLE
22.1 Accidental cameras22.1.1 The accidental pinhole: a window is a camera22.1.2 The accidental pinspeck: the anti-pinhole22.1.3 The occluder as a crude lens, and recovery as deconvolution22.1.4 Corners and doorways: an edge that resolves the hidden room22.1.5 Where else the world hides a camera22.2 Reflections in the eye22.2.1 The cornea as a catadioptric mirror22.2.2 The geometry: from a corneal pixel to a direction in the world22.2.3 What the recovered reflection is good for22.2.4 Eyes for relighting22.3 Motion and video magnification22.3.1 The Lagrangian precursor: track, then exaggerate22.3.2 Eulerian video magnification: amplify the time series at each pixel22.3.3 Why amplifying brightness amplifies motion22.3.4 Phase-based magnification: move the motion into phase22.3.5 What it reveals: vital signs, structures, materials, modes22.4 Visual microphone22.4.1 From sub-pixel motion to a sound waveform22.4.2 Bandwidth: high-speed cameras and the rolling-shutter trick22.4.3 How good is the copy? The object's frequency response22.4.4 The active cousins: laser vibrometry and interferometry22.5 Corner camera22.5.1 The edge as a one-dimensional aperture22.5.2 From a faint gradient to a usable signal22.5.3 What the corner can and cannot tell you22.6 Active non-line-of-sight22.6.1 Third-bounce geometry and time-of-flight22.6.2 The hardware: photographing light in flight22.6.3 From back-projection to fast, exact inversion22.6.4 The trade, stated plainly22.7 Passive non-line-of-sight22.7.1 The occluder is what makes it solvable22.7.2 A deconvolution where the lens is unknown22.7.3 One dimension: the corner camera22.7.4 Two dimensions from a single photo: computational periscopy22.7.5 Active versus passive, the ledger22.8 Mm-wave, wifi22.8.1 Why radio walks through walls22.8.2 Time of flight, again: radar is NLOS with a longer wave22.8.3 From a radio smear to a human skeleton: the learned map22.8.4 What it sees, and what it costs us23 ADJACENT FIELDS AND APPLICATIONS
23.1 Optical computingoutline 23.2 Astrooutline 23.2.1 Extreme long exposure23.2.2 Tracking, stacking, and selection (pointers)23.3 X-rayoutline 23.4 Medicaloutline 23.5 Microscopyoutline 23.6 Mm-waveoutline 23.7 Music, soundoutline 23.8 Fluorescenceoutline 23.9 Opto-acousticoutline 23.10 Ultrasoundoutline 23.11 Aerial imagingoutline 23.12 Computer visionoutline 23.13 Robotics, drivingoutline 24 HUMAN FACTORS
24.1 Human factors and the art of photography24.1.1 Make better photos24.1.2 Typical shooting scenarios24.1.3 Macro photography24.1.4 Special effect photography24.1.5 Fun artsy stuff24.1.6 Perception of art24.2 Ethics of computational photography24.3 Computational models of perceptionoutline 24.3.1 Spatial (and spatio-temporal) vision24.4 User studiesoutline 24.5 Accessibility: photography by and for blind usersoutline 24.5.1 Blind camera — capture without a sighted operator24.6 The social and personal practice of photographyoutline 25 IMAGE FORENSICS AND AUTHENTICATION
25.1 Image Forensicsoutline 25.1.1 The problem and the threat model25.1.2 Sensor and pipeline traces: PRNU, CFA, and noise25.1.3 Compression, geometry, and metadata forensics25.1.4 Deepfakes, GAN/diffusion fingerprints, and learned detection25.1.5 Why forensics is evidence, not proof — and the hand-off to provenance25.2 Authentication and Provenance (C2PA)25.2.1 From detection to attestation: trustworthy cameras and watermarking25.2.2 C2PA and Content Credentials: the standard25.2.3 AI disclosure, watermarking, and the regulatory push25.2.4 Limits, critiques, and the forensics partnership26 SYSTEMS
26.1 Programmable and modular camerasoutline 26.2 Image processing librariesoutline 26.3 Lightroom-style raw developersoutline 26.4 Photoshop-style editorsoutline 26.5 Networking and image transportoutline 26.6 Photography programming on phonesoutline 27 PERFORMANCE ENGINEERING AND HALIDE
27.1 8-bit and fixed-point arithmetic27.1.1 Integer versus fixed-point: where the binary point sits27.1.2 Rounding, dithering, and the banding trap27.1.3 Saturation and the width of the accumulator27.1.4 fp16 versus bf16: the exponent–mantissa bargain27.1.5 int8 quantization for neural inference27.1.6 When float is non-negotiable27.2 Algorithmic speedups27.2.1 Separability: a 2-D pass for the price of two 1-D passes27.2.2 Recursive / IIR filters: a running state, cost independent of radius27.2.3 Integral images / summed-area tables: any box sum in four lookups27.2.4 Fast median filters: a sliding histogram for a constant-time median27.2.5 Pyramids and multiscale: do the large-scale work on small images27.2.6 Discretize the range to accelerate non-linear filters27.2.7 Downsampling and edge-aware upsampling27.2.8 Where this goes next27.3 Automatic search for fast methods27.3.1 The problem is a curve, not a point27.3.2 Why search beats a hand-shrunk CNN27.3.3 Ma et al. 2022: searching structure and parameters together27.3.4 Three axes of "search instead of design"27.3.5 The broader family, and what it costs27.3.6 Where this goes next27.4 Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast27.4.1 Why moving data, not doing math, is the bottleneck27.4.2 The memory hierarchy and locality27.4.3 The roofline: is my kernel compute- or memory-bound?27.4.4 The forms of parallelism27.4.5 What it takes to leverage a modern machine27.4.6 Why photography is hard for the machine27.4.7 Where this goes next27.5 Hardware backends: GPU, NPU, DSP27.5.1 GPU: the data-parallel workhorse27.5.2 The neural accelerator: NPU and TPU27.5.3 DSP: the real-time control loop27.5.4 ISP, FPGA, and ASIC: the fixed-function end27.5.5 The heterogeneous SoC, and the spectrum to carry away27.5.6 On-device ML runtimes, and why the work stays on the phone27.6 Halide: Decoupling Algorithms from Schedules27.6.1 What it means to separate the algorithm from the schedule27.6.2 Why image pipelines are uniquely hard to optimize27.6.3 The split was always there: done by hand27.6.4 The scheduling space27.6.5 Auto-scheduling: letting the compiler search27.6.6 Results, impact, and reach27.6.7 Gradient Halide: differentiating the pipeline27.6.8 Where this goes next27.7 Halide programming27.7.1 The three nouns: `Func`, `Var`, `Expr`27.7.2 A first image pipeline: brighten, then blur27.7.3 Reductions: `RDom`, sums, and histograms27.7.4 The default schedule, and seeing the loops27.7.5 Scheduling the loops within a stage: `reorder`, `split`, `tile`, `vectorize`, `unroll`, `parallel`, `fuse`27.7.6 The heart of it: producer–consumer granularity (`compute_at`, `store_at`)27.7.7 The schedule ladder, with numbers: ten times faster from one line27.7.8 Boundaries and bounds27.7.9 Same algorithm, new machine: the GPU27.7.10 Letting the compiler schedule: the auto-scheduler27.7.11 The development loop: correctness, measurement, and benchmarking hygiene27.8 Efficient neural network inference27.8.1 Quantization: fewer bits per weight27.8.2 Pruning: fewer weights27.8.3 Knowledge distillation: a small student, a big teacher27.8.4 Low-rank and tensor factorization27.8.5 Efficient architectures: cheapness designed in27.8.6 Neural architecture search: let the machine design it27.8.7 Hardware-aware deployment and the roofline27.8.8 Where this leaves the part28 CONCLUSIONS, DISCUSSION
28.1 Recap in contextoutline 28.1.1 Modern phones, multiple apertures, pano, HDR+28.1.2 Recap: a modern mirrorless camera28.1.3 Recap: a modern cell phone multi camera28.1.4 Lightroom28.1.5 Photoshop28.2 Why phones are so good (at photography)28.2.1 Computation beats glass28.2.2 The whole pipeline is co-designed28.2.3 Machine learning and data at scale28.2.4 Many small cameras for one big one28.2.5 The human and system advantages28.2.6 The hard caveats: physics still wins where it must28.2.7 The throughline29 BACK MATTER
29.1 Bibliography 29.2 Glossary 29.3 Acronyms 29.4 Term index 30 APPENDICES
30.1 Refreshers30.1.1 Linear algebra30.1.2 Calculus: derivatives, gradients, integrals30.1.3 Optimization and regression30.1.4 Probability and information theory30.1.5 Machine learning and deep learning30.1.6 Programming: Python, C++, and PyTorch30.2 Problem Set 0 — Environment and C++ basics30.2.1 Summary30.2.2 Installation and Environment Setup30.2.3 C++30.2.4 Submission30.3 Problem Set 1 — Image class, point operations, and color30.3.1 Summary30.3.2 The Image Class30.3.3 Brightness and Contrast30.3.4 More Image Class Methods30.3.5 Colorspaces30.3.6 Spanish Castle Illusion30.3.7 White Balance30.4 Problem Set 2 — Convolution and the bilateral filter30.4.1 Summary30.4.2 Smart Accessor30.4.3 Blurring30.4.4 Denoising using Bilateral Filtering30.4.5 Extra credit30.4.6 Submission30.5 Problem Set 3 — Denoising and demosaicking30.5.1 Summary30.5.2 Denoising from a sequence of images30.5.3 Demosaicing30.5.4 Edge-based green30.5.5 Red and blue based on green30.5.6 6.865 only (or 5% Extra Credit): Sergey Prokudin-Gorsky30.5.7 Extra credit (maximum of 10%)30.6 Problem Set 4 — High dynamic range30.6.1 Summary30.6.2 HDR merging30.6.3 Tone mapping30.6.4 Extra credit (10% max)30.7 Problem Set 5 — Resampling, warping, and morphing30.7.1 Summary30.7.2 Resampling30.7.3 Warping and morphing30.7.4 Extra credit30.8 Problem Set 6 — Homographies and manual panoramas30.8.1 Summary30.8.2 Class Morph30.8.3 Homogeneous Coordinates30.8.4 Linear Algebra30.8.5 Warp and Image with a Homography30.8.6 Compute Homography from 4 Pairs of Points30.8.7 Bounding boxes30.8.8 Extra Credit (up to 10% total)30.9 Problem Set 7 — Automatic panoramas30.9.1 Summary30.9.2 Previous Problem Set Code30.9.3 Class Morph30.9.4 Harris Corner Detection30.9.5 Descriptor and correspondences30.9.6 RANSAC30.9.7 Automatic panorama stitching30.9.8 Blending30.9.9 Mini planet30.9.10 6.8370: Stitch N Images (6.8371: Extra Credit 5%)30.9.11 Make your own panorama30.9.12 Extra credits (10% max)30.10 Problem Set 8 — Non-photorealistic rendering30.10.1 Summary30.10.2 Paintbrush splatting30.10.3 Painterly rendering30.10.4 Oriented painterly rendering30.10.5 Your image30.10.6 Paper Review (6.865 only)30.10.7 Extra credits30.11 Problem Set 9 — Make-your-own, video, and ethics30.11.1 Summary30.11.2 Make Your Own Assignment30.11.3 Ethical issues in computational photography30.11.4 Assignment Lists30.12 EXIF and image metadata30.12.1 What EXIF is30.12.2 The fields, grouped by what they describe30.12.3 How far to trust it30.12.4 Privacy: the metadata that follows the picture30.12.5 Reading and writing EXIF30.13 DNG: the Digital Negative30.13.1 What DNG is, and the problem it solves30.13.2 Inside the container30.13.3 What the raw payload looks like: mosaic vs linear30.13.4 The color recipe: matrices, profiles, and white balance30.13.5 Opcodes: corrections the decoder must apply30.13.6 Compression, and embedding the original30.13.7 Where you meet DNG: adoption and relatives30.13.8 Trade-offs, and DNG's relation to EXIF30.14 Rendering a raw DNG30.14.1 The pipeline, step by step30.14.2 Where Lightroom's "look" actually comes from30.14.3 What we would ask an Adobe engineer to check30.14.4 Two kinds of DNG, and the special cases30.14.5 Easy mistakes (most of which we made)30.15 Datasets30.15.1 Classification and features30.15.2 Super-resolution30.15.3 Deblurring and restoration30.15.4 Denoising30.15.5 HDR and tone mapping30.15.6 Retouching and enhancement30.15.7 Depth and motion30.15.8 Light fields30.15.9 Color and white balance30.15.10 Faces30.15.11 Inpainting, segmentation, and matting30.15.12 Image quality30.16 A camera-feature wish list30.16.1 Exposure, ISO, and dynamic range30.16.2 Bracketing more than exposure30.16.3 Focus and depth of field30.16.4 Computational raw and the sensor30.16.5 Motion data, metadata, and workflow30.16.6 Panorama and multi-shot30.16.7 The interface and the ecosystem30.17 How this book was created30.17.1 Two documents, not one30.17.2 Compiling a section30.17.3 Figures as code30.17.4 Generative imagery: cover art and 3D30.17.5 Verification and review30.17.6 Keeping a long book coherent30.17.7 The toolchain30.17.8 What the machine did, and what it did not30.17.9 Who wrote what: a per-part estimate30.18 The course tutor: a local, book-grounded AI teaching assistant30.18.1 What it is, and what it is for30.18.2 Local-first30.18.3 Grounded in the book: retrieval-augmented generation30.18.4 It links, it shows equations, it shows figures30.18.5 Two front-ends, one core30.18.6 What the instructor sees30.18.7 Privacy and candor30.19 The semi-automatic grading system30.19.1 The shape (to be confirmed)30.19.2 Automatic versus human (to be confirmed)30.19.3 To be filled in (from the instructor)30.20 Under the hood: prompts, patterns, and verifiers30.20.1 Prompt patterns that made it work30.20.2 The verifier suite30.20.3 Why this is the interesting part30.21 Reading a Lytro light field30.21.1 The container: the LFP/LFR format30.21.2 The pipeline, step by step30.21.3 From hexagonal lenslets to a uniform grid30.21.4 Why a white image30.21.5 Open questions and where we approximate30.21.6 Easy mistakes (most of which we made)30.22 File conversion tools30.22.1 JPG → PNG, without an alpha channel30.22.2 DNG → a linear PNG30.22.3 A Lytro capture → a Stanford-style light-field archive30.22.4 How they run in the browser30.22.5 Easy mistakes30.23 The interactive figures — a prompt-by-prompt making-of30.23.1 Summary30.23.2 Lorentz resonance — `fig-lorentz-resonance` (Figure 2.1.8)30.23.3 Rainbow droplet — `fig-rainbow-droplet` (Figure 2.1.14)30.23.4 Diffraction wave simulation — `fig-diffraction-wave-sim` (Figure 2.1.29)30.23.5 Exposure-triangle simulator — `fig-exposure-triangle-sim` (Figure 2.11.5)30.23.6 Repeated quantization — `fig-repeated-quantization` (Figure 3.1.4)30.23.7 JPEG generation loss — `fig-jpeg-generation-loss` (Figure 3.17.9)30.23.8 Exposure round-trip (JPEG) — `fig-exposure-jpeg-roundtrip` (Figure 3.17.10)30.23.9 1-D sampling pipeline — `fig-sampling-1d-demo` (Figure 3.10.4)30.23.10 2-D sampling pipeline — `fig-sampling-2d-pipeline` (Figure 3.10.6)30.23.11 Mitchell–Netravali bicubic (B,C) — `fig-bicubic-bc` (Figure 3.11.13)30.23.12 Rotation resample challenge — `fig-rotation-resample-interactive` (Figure 3.11.18)30.23.13 Mini-Lightroom — `fig-mini-lightroom` (Figure 3.18.4)30.23.14 Poisson blending — `fig-poisson-blend` (Figure 5.1.5)30.23.15 Bilateral grid (3-D) — `fig-bilateral-grid-3d` (Figure 5.2.15)30.23.16 Perspective montage — `fig-perspective-montage` (Figure 6.6.13)30.23.17 Beier–Neely morph — `fig-beier-neely-demo` (Figure 6.3.4)30.23.18 Live face landmarks — `fig-face-landmarks-live` (Figure 7.10.2)30.23.19 Lens optimizer — `fig-lens-optimizer-demo` (Figure 9.4.4)30.23.20 Full auto-panorama — `fig-pano-stitch` (Figure 10.7.9)30.23.21 CLIP-IQA curation — `fig-clip-iqa` (Figure 11.3.2)30.23.22 Photobio time-lapse — `fig-photobio-demo` (Figure 11.7.2)30.23.23 Refocus shift geometry — `fig-refocus-shift-geometry` (Figure 13.4.2)30.23.24 Chroma key — `fig-chroma-key` (Figure 8.11.7)30.23.25 Portrait-lighting simulator — `fig-portrait-lighting-sim` (Figure 2.11.26)30.23.26 Aberration explorer — `fig-aberration-explorer` (Figure 9.1.6)30.24 Interactive demo index30.24.1 Sharing, linking, and embedding a demo30.24.2 Introduction30.24.3 Fundamentals — light, optics, sensors, color30.24.4 Basic image processing and the ISP30.24.5 Computational tools — machine learning and diffusion30.24.6 Edges matter — gradient-domain and edge-preserving30.24.7 Warping and morphing30.24.8 Matching pixels across space and time30.24.9 Single-image computational photography30.24.10 Optics, lenses, and aberration correction30.24.11 Multiple-exposure imaging — HDR and panoramas30.24.12 Many images and photo collections30.24.13 Video30.24.14 Light fields and plenoptic cameras30.24.15 3-D and depth30.24.16 Appendices and end matter30.24.17 Not yet placed