☰ ‹
draft · v0.1.226 INTRO From Digital to Computational Photography Computation is the new optics The goals of computational photography What makes a technique useful A field in transition: from hand designed algorithms to learning to generative AI Does photography still matter in the age of generative AI? A field at a crossroads It's not just about photography How to read this book Machine-readable by design How this book was made What programming language for computational photography? On the desktop: Python, C++, and Halide In the browser: JavaScript and the web stack On the phone: Android and iOS Libraries Vibe coding Problem sets FUNDAMENTALS OF IMAGING Light and physics Light: rays, waves, and the spectrum How light is created Light power and brightness Summing coherent vs. incoherent light Reflection, refraction, and what happens at a surface Polarization The color of objects: illumination times reflectance The BRDF and the look of materials Illuminants Radiometry: radiance, irradiance, exposure, and falloff Global illumination Wave effects, diffraction, and the diffraction limit Perceptual color and trichromatic vision Visual system anatomy: Color Opponent process and the multistage model So what are the primary colors? Human Vision Light adaptation Lightness constancy Color constancy Contrast Spatial vision Temporal vision Attention and eye movements The visual system makes things up Animal eyes The optics: many ways to form an image The color vision: opsins remixed Measuring and encoding color analysis vs synthesis and non-orthogonality Measuring color Chromaticity diagram Linear vs Gamma vs. log encoding RGB color spaces Non-linear perceptually-uniform color space HSV, HSL, and cylindrical color spaces Reproducing color Skin tones Pinhole image formation and linear perspective Pinhole imaging and the perspective projection Homogeneous coordinates Camera in a general configuration Intrinsics, extrinsics, and what cropping really does What perspective preserves, and what it destroys Wide-angle distortion: spheres bulge and faces stretch at the edges Photography with focal length: framing, magnification, and compression Depth, ray length, and unprojection Lens image formation Single lenses: refraction and Snell's law Thin lens optics Aperture and the f-number Image measurements as integrals The plenoptic function The pixel integral Exposure values (EV) and stops Splitting the integral Depth of field The circle of confusion Depth of field versus depth of focus Choosing the maximum circle of confusion The double cone, and the near and far limits Hyperfocal distance The surprising invariance Background blur beyond the focus plane Defocus is not a blur of the image Sensor size scales depth of field Motion blur Blur from subject motion Blur from camera shake How shift-invariant is camera-shake blur? Which dominates: translation or rotation? The hand-holding rule of thumb Shake from the shutter and mirror Panning: fighting subject motion by moving the camera Freezing motion with light Sensors: photosites, CCD vs CMOS The photoelectric effect Photon to number: the photosite Spectral sensitivity: matching the eye, catching photons, and white balance Beyond the visible spectrum: near-infrared, thermal, and ultraviolet Analog-to-digital conversion CCD versus CMOS Time integration: the exposure Shutters A taste of modern sensor tricks The impact of sensor size How far sensors have come Sensing color: multiplexing strategies Multiplexing in time Multiplexing in space: the color filter array Multiplexing across separate sensors: the beam-splitter Multiplexing per pixel by dispersion: color routers and nano-prisms Multiplexing per pixel by tunable absorbers: quantum dots Multiplexing in depth: stacked photodiodes Hybrid strategies Beyond trichromatic capture: full spectrum and multispectral Noise, signal-to-noise ratio and dynamic range Noise sources Noise variance is affine in pixel brightness The algebra of noise Signal-to-noise ratio: it's about ratios Dynamic range Regimes, and how close we are to the limits Imaging as a linear system Linear systems from a billboard to an image Linear systems from ray space (light fields) to an image Space–time: imaging across time When linearity breaks down Where this is going: invertibility Imaging as an inverse problem Linear algebra: how hard is this problem? Priors and the manifold of natural images Probability: the Bayesian view Optimization, inference, loss When you can design the operator — and a teaser of information theory Recap: what makes imaging hard, and where it's going Harder inverse problems: factorization Limitations of the medium Compensation, accentuation, conflict The picture is flat: depth and its cues The picture is static: time and motion One viewpoint, and a finite frame The contrast is limited: dynamic range and gamut Resolution: the limitation we are overcoming Displays The range of displays Display technology HDR displays and dual modulation The film movie projector Printing Why the display characteristics matter Distance, resolution and acuity Sharpening for the display: size and distance Light level viewing conditions Robustness of perspective to the viewer's viewpoint Gamut and gamut mapping Color management, ICC, and industry standards PHOTOGRAPHY Exposure Basic photography: exposure settings — shutter, aperture, and ISO Metering: from printed guides to the sensor f-stops versus T-stops From exposure to photometry Exposure modes, UI, and auto-ISO Lenses Lenses and focal length Aperture and the diaphragm Focusing the lens Image stabilization Keeping the lens clean Lens filters: polarizers, ND, and graduated ND Focus, autofocus, and depth of field Manual-focus aids: making the focal plane visible Cameras UI and display How the viewfinder works Anatomy of a modern full-frame interchangeable-lens camera Anatomy of cell-phone cameras Camera versus phone Types of cameras Cameras beyond photography A camera's other sensors Video Cinema lenses Illumination and the flash Natural illumination Flash Flash metering Indirect illumination and bounce flash Multiple-point lighting Traditional and Digital Darkroom The traditional darkroom The digital darkroom Programmability, or the lack thereof Photographs are usually not passive objective recordings Faithful is not the same as realistic Choosing the instant, choosing the viewpoint Speed, aperture, and focus: the photographer rules Filters and lighting: building the scene's light The darkroom: tone, dodging, and burning Make-up and retouching: authoring the subject Getting closer — systematic capture, and the biases that remain BASIC IMAGE PROCESSING AND ISP Image representation An image is an array Float vs 8-bit vs more bits Stochastic Quantization and Dithering How the array sits in memory Pixel coordinate conventions Alpha and extra channels Video and more dimensions Pixel lookups and boundaries Beyond the pixels: basic metadata and EXIF Three kinds of operation What the numbers mean What the numbers actually mean: encoding What the numbers actually mean: color spaces Developing, Testing and Debugging The workflow: build, run, look at the picture Principles Test on inputs you can verify by hand Finding good real test inputs Per-algorithm debug recipes Crashes and bounds Vibe coding: writing image code with an LLM Point operations Three kinds of operation The point-operation curve Exposure: a multiply in linear light Brightness vs. exposure Contrast: steepen about a pivot Black point, white point, and levels The general tone curve Basic color enhancement: saturation and vibrance Converting to black and white Lookup tables Video color grading Local point operations Histograms the histogram the histogram depends on the encoding space a histogram is a sampled estimate histogram equalization histogram matching constraining the slope: Ward's histogram-based tone mapping Global tone mapping The dynamic-range problem Two families: global and local Global tone mapping: one curve for everything Histogram equalization and Ward's bound Local tone mapping and the halo problem: why naive local fails Why log space pays off The analog ancestor: the Zone System HDR on every screen: the gain map Where this is going Beauty curves: the camera "look" Why scene-linear looks dull The shape of the curve Highlights: roll off to white, not to a color Saturation, and the rest of the "look" The discipline Neighborhood operations and convolution Motivation: blur and sharpen Convolution 101 The flip: where-from vs. where-to Properties: the impulse, normalization, symmetry, commutativity Nitty-gritty: finite images A blur zoo Separability Gradient and oriented filters Is imaging blur usually a shift-invariant convolution? Where this goes next Sharpening Linear sharpening Why it's called "unsharp mask" Sharpen in gamma, linear, or log? Non-linear sharpening Fourier Images as vectors in a high-dimensional space why Fourier Definition: one coefficient per wave Sines are the eigenvectors of convolution The 2-pixel example Reading an image's Fourier transform A small bestiary of transforms The fine print: two limitations of Fourier Windowing: the price of measuring a real spectrum Application preview: can we deblur, given the blur? Sampling, downsampling, and aliasing Aliasing: a high frequency in disguise Nyquist and the sampling theorem In two dimensions, aliasing has a direction Sampling in the frequency domain: spectral replicas Seeing it: sampling and pre-filtering, hands on The ideal reconstruction filter: sinc, and why it is unreachable Resampling and upsampling Domain operations: moving pixels around Start with scaling up The naive idea, and why it leaves gaps Loop over the output, use the inverse transform Nearest neighbor Linear interpolation, in 1-D first Bilinear, in 2-D The convolution perspective Better kernels: bicubic and Lanczos Upsampling vs. downsampling: scale the kernel Prefiltering when the transform isn't a clean scale Beyond Nyquist-limited reconstruction Where this goes next Pyramids and wavelets Halfway between space and frequency The idea: process an image at many resolutions The Gaussian pyramid The Laplacian pyramid Reconstruction: an exact encoder and decoder Each level is a frequency band What the bands look like on a real image Candid limitations Cousins: wavelets and steerable pyramids Pyramid blending Other applications Multiresolution as a recurring theme Image metrics Full-reference vs. no-reference why not just L2? PSNR SSIM VDP and HDR-VDP learned metrics The right metric depends on the task Denoising basics what is noise? denoising by averaging multiple shots denoising from a single image Spatial averaging and its limits The bilateral filter: averaging by affinity Denoising in the pyramid: coring Denoise color more than brightness noise estimation The limits of denoising Demosaicking Reminder: the Bayer mosaic The task: full RGB at every pixel The naive approach: interpolate each channel on its own Why naive interpolation zippers: averaging across an edge Doing better: edge-directed interpolation The harder half: red and blue, and color fringing Green-based demosaicking: interpolate the color difference Classic (non-learning) demosaicking: the general strategy Related: the optical anti-aliasing filter Quad-Bayer sensors: remosaic before demosaicking Fuji X-Trans: a larger, irregular CFA Beyond hand-tuned: joint denoising and learned demosaicking Cross-reference: other ways to sense color Where this sits in the pipeline Auto-exposure and auto white balance Auto-exposure: metering white balance and color constancy Automatic white balance The limits of white balance, and CRI File formats and compression The big picture: none, lossless, lossy Data versus metadata: EXIF PNG: the format we read from JPEG: compression by perception RAW files: before the cooking HDR formats: more than 8 bits Modern formats Other formats in passing: TIFF and GIF Recap ISP, non-destructive editing: A basic ISP Pipeline design and tuning The ISP, evolving: traditional → learned → generative Recap 2: non-destructive editing COMPUTATIONAL TOOLSLinear Inverse Problems and RegressionBlur is linear, so deblurring is inversionImages as vectors, and the notation overloadRegression: deblurring as least squaresMatrices without forming matrices: gradient descent and conjugate gradientEfficient solversEfficient solversMatrix-free, iterative, multiscale: the situationGradient descent and conjugate gradient (recap + preconditioning)Multigrid: coarse-to-fine on a pyramidThe FFT solver: diagonalize when you canWhich solver wins — and why this is the foundation of the partInformation theoryThe basics, at a high levelCompression: the clearest applicationThe imaging system as a channelInformation-theoretic quantities as objectivesThe modern revival: learned compression and generative modelsCousins, not childrenWhere it is least relevant: aesthetics and perceptionFundamental limits of inverse problemsThe question: what can no algorithm recover?The SVD picture: null space, near-null, and the noise explosionInformation bounds: a floor no estimator beatsThe resolution–noise tradeoff, and how a prior inventsFundamental versus practical, and coded capture as the escapeMachine learningThe framing: learned operators replace hand-designed onesThe data story: synthetic data, noise models, datasetsDeep learningLow-level learned operators (pixel-to-pixel)Mid- and high-level learned predictorsGenerative models: image-to-image translationLearned perceptual metrics and lossesGenerative AI and diffusionThe framing: generation is learning and sampling a prior $p(x)$Diffusion: generation as iterated denoisingConditioning: text, images, and controlPosterior sampling: generative priors for inverse problemsOther generative families, in briefCaveats and ethicsImage priorsA catalog of priors: from smoothness to learnedShift and scale invarianceAnisotropy: sparsity in a single directionLinear color modelsDenoising as a universal prior: Plug-and-Play and REDDiffusion is iterated denoising (the continuous limit)Score-based models, the thin manifold, and imperfect priorsEDGES MATTERBilateral filteringMotivation: local tone-mapping haloesBilateral filterCross / joint bilateralBilateral gridBilateral-grid learning (HDRnet)Non-local meansEdge-preserving optimization — colorizationThe shift: filtering becomes optimizationColorization: the canonical demonstrationWriting down the energySolving it: a sparse linear system we already ownOne affinity, two faces: the matting Laplacian connectionLocal Laplacian filtersThe wish, and why the bilateral leaves a haloWhat a Laplacian pyramid hands youThe mechanism: a remapping recomputed per output pixelOne curve, three edits: enhancement, tone mapping, inverse tone mappingContrast with the bilateral base/detail splitThe cost, and the fast versionMatting LaplacianThe matting problem: compositing, and why it is under-determinedThe color-line assumption: α becomes linear in colorClosed-form matting: the matting LaplacianSpectral matting: eigenvectors of the matting LaplacianAn alternative matting LaplacianBeyond matting: spatially-varying white balanceGuided image filteringThe key idea: the output is a local line in the guideFitting the line, and the one knobWhy it is fast: O(N) regardless of window sizeNo gradient reversalReading the affinity back outWhere it is usedLocally adaptive regression kernel (LARK)Change the verb: from averaging to fittingSteering the kernel to the local structureThe regression flavor of the affinityLARK as a structure descriptorPoisson image editingEdit gradients, not pixelsSeamless cloning: the headline applicationReconstruction = solving the Poisson equationPoisson vs. pyramid blendingAdvanced techniquesSeam optimizationFind the non-edges: least-noticeable cutsIntelligent scissors / live-wire: the cut as a least-cost pathSnakes (active contours): the continuous cousinGraph cut: globally optimal boundaries by min-cut / max-flowGrabCut: interactive cutout from a single rectangleNormalized cuts: the spectral relaxationSeam carving: content-aware resize by DPTime-lapse via DP: a seam through timeVideo textures: looping a seam in timeGraphCut textures & photomontage: cut, then blendWhere this connects: MRFs and beyondRecap: which edge-aware technique when?The three relationships to edgesPros, cons, and cost: when to reach for eachA worked chooserWARPING, MORPHING, AND LAGRANGIAN APPROACHESWarpingWarping is a domain transform: move where, not whatForward vs inverse warping: why output-driven winsResampling: a quick reminder (developed in BASIC)Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladderSpecifying the warp II: free-form warps from sparse correspondencesLiquify: the warp with a paintbrush bolted onAntialiasing for complex transformsWhen the footprint stops being a square: the warp JacobianEWA: the elliptical weighted averageFeline and anisotropic MIP probing: EWA on a GPU budgetOther footprint integrators: summed-area tables and ripmapsFootprint estimation, clamping, and when to reach for itMorphingWhy a cross-dissolve isn't enough: the ghosting motivationTwo interpolations: domain (shape) and range (color)The morphing recipe (combine both)Field morphing: the Beier–Neely line-pair warpMesh / triangulation morphing (the alternative warp)View morphing: the geometrically-correct in-between of two viewsRecap and significanceMorphable modelsStep 1: dense correspondence, the precondition for averaging shapesStep 2: Procrustes alignment, factor out pose so PCA sees shapeStep 3: PCA, a mean and a basis of eigen-deformationsStep 4: shape and appearance are separate basesStep 5: fitting a new photo, analysis-by-synthesisStep 6: once fitted, edit by moving in the spaceStep 7: from linear PCA to neural priorsRecap and significanceShape-preserving warpingWhy plain interpolation shears: the rigidity gapAs-rigid-as-possible (ARAP) manipulationLinear blend skinning and why the weights decide everythingBounded biharmonic weights (BBW)ARAP interpolation: rigid-as-possible morphing between two posesMoving least squares: the simpler cousinRecap and significancePerspective distortion and its correctionKeystoning is projection, not a lens flawThe fix is a homography: re-render the façade fronto-parallelThe optical alternative at capture: tilt-shift / ScheimpflugCropping wide-angle photographs: recenter with a homography, don't just cropA different perspective distortion: wide-angle portraits, and a content-aware fixBeyond faces: correcting arbitrary objects (MaDCoW)A montage has no single viewpoint: a family in a boxRecomposing perspective after capture: computational zoomThe catch: resampling cost, and "only a plane rectifies exactly"Where this sits: one map, then transportMATCHING PIXELS AND HUMANS ACROSS SPACE AND TIMEBrute forceWhy you must align firstBrute-force translational alignment (SSD / NCC)Phase correlation: the whole shift from one FFTCoarse-to-fine alignment on a pyramidSub-pixel matchingRefining the cost: fit a parabolaRefining in the frequency domain: phase correlationRefining by gradient: the Lucas–Kanade stepWhat breaks it: peak-locking, texture, and noiseOne tool, everywhere downstreamSparse matchingWhere to look: corners, and the structure tensorInvariance: surviving scale and rotationDescribing a neighborhood: SIFT and its zooMatching: nearest neighbor in descriptor spaceFeature trackingTracking vs dense flow: sparse-but-long vs dense-but-shortKLT = Lucas–Kanade, per feature, iterated over timeGood features to track = where the structure tensor is well-conditionedWhat breaks long-term tracking: drift, appearance change, occlusion, and re-detectionModern point trackers (briefly)An application: synthetic motion blur from a trackRobustness: the ratio test and RANSACThe ratio test: reject ambiguous matches before fittingRANSAC: fit from a minimal random sample, score by consensusVariants and degeneracyDeep learning approaches to sparse matchingLearned local features: SIFT's job, done by a networkLearned matching — reason about the whole set at onceThe 3-D-aware turn: pointmaps subsume matchingFast matchingApproximate nearest neighbors for sparse descriptorsPatchMatch: randomized dense correspondence by propagationFast high-dimensional matching by random projectionOptical flowWhat optical flow is: and whether it is even well-definedBrightness constancy and the optical-flow constraintThe aperture problemLucas–Kanade: local constant-flow least squares (and the structure tensor)Horn–Schunck: global smoothness regularizationLarge motion: coarse-to-fine warpingLearned flow: RAFT (neuralize the classical pipeline)Deep learning approaches to optical flowThe unrolling principle: neuralize the classical solverCost volumes and warping inside the netRAFT and the recurrent updateFace trackingDetecting the faceLandmarks: pinning down the featuresTracking across timeLifting to 3-D: the morphable modelRecognition, and the dark sideWhich library to useBody pose estimationTop-down vs bottom-upOn-device, real timeLifting to 3-D: parametric bodiesWhich library to useSINGLE IMAGE COMPUTATIONAL PHOTOGRAPHYDenoisingA reminder: bilateral filtering and wavelet/pyramid shrinkageBM3D: group similar patches and filter them togetherLearning to denoiseSuper-resolutionWhat problem super-resolution solves (and why it's ill-posed)Scenarios: single-image, burst, and hybrid space–timeReconstruction vs hallucination: measured detail vs invented detailDemosaicking and joint reconstructionThe pipeline problemFlexISP: one energy, one priorLearning the joint prior: the founding formulationModel-based and unrolled networksLearning where the hard cases areCo-designing the mosaic: a learnable sensorThe end-to-end learned ISPBetter backbones: CNN → transformer → diffusion priorExotic mosaics: learned remosaickingWhere do the labels come from?Correcting chromatic aberration jointlyRecapNon-blind deblurringDeblurring in the presence of noise: why naive inversion failsThe Wiener filter — the regularized, noise-aware inverseSparse gradients: the prior that keeps edges sharpBlind deblurringBlind deblurring: estimating the kernel and the imageA more realistic blur model: spatially-varying (camera-shake) blurEngineering the aperture: depth and all-focus from a coded maskDehazingDehazing as a prior-driven inverse problemDifferentiable image pipelines and algorithm optimization (Halide)Mixed-lighting white balanceWhy a global gain must failEstimating the per-pixel mixture: an under-determined inverse problemCorrecting each region for its own lightWhat came afterInpainting, texture synthesis, and object removalInpainting as filling unmeasured pixels — the spectrum of priorsPDE / diffusion-based inpaintingTexture synthesis (Efros–Leung; Efros–Freeman quilting)Exemplar inpainting: clone, healing brush, and object removal (Criminisi)Data-driven scene completion (Hays & Efros)Deep inpainting (context encoders → partial/gated conv → diffusion)Highlight / specular recoveryEpitomes — a compact patch modelPatch matchThe nearest-neighbor field, and why exhaustive search is the bottleneckPatchMatch — randomized correspondence Barnes et al. 2009Applications: hole filling, retargeting, reshuffleShift-Map image editing Pritch et al. 2009 — editing as graph-cut labelingColorizationOne channel in, three channels outA spectrum of priors: scribbles, references, and learned modelsThe multimodality trap: why naïve colorization goes muddyClosing the loop: learned priors with a human's hintsPlausible is not correctWhere it sitsCompositing, segmentation and mattingCompositing and alpha channelsSegmentation: cutting the object outThe fundamental matting equationBlue/green-screen matting and chroma keyingTraditional matting approachesDeep-learning mattingGenerative mattingHarmonization and where the blends liveOptical effects beyond alphaIllumination related effects in a single imageIntrinsic images: the unifying frame ($I = R\cdot S$)Multiple-light / spatially-varying white balanceReflection removal — pulling apart a transmission and a glass reflectionShadow detection and shadow removalSpecular-highlight removal / "fake polarization" (the dichromatic model)Tone MappingGlobal vs local, re-hashedA smarter global curve: histogram adjustmentA taxonomy of local methodsThe darkroom ancestor: dodge & burn and the Zone SystemStyle transferClassical style transfer: patches and statisticsNeural style and feed-forward stylizationStyle transfer as image-to-image translationNon-photorealistic renderingWhat NPR is for, and the one ideaStroke-based / painterly rendering (and the brush p-set)Edge-preserving abstraction: bilateral + Difference-of-GaussiansExample-based stylization and the bridge to neural styleRegion-based stylization: stained glass, low-poly, mosaicsArtistic screening and halftoningCOMPOUND LENSES, AND ABERRATION CORRECTIONAberrations and optical challengesTaxonomy of challengesSpherical aberrationComaAstigmatismField curvatureChromatic aberrationRadial distortionWave effects and diffractionVignettingFlare and coatingAberrations correctionThe two families of cureCorrection in glassComputational correctionRadial distortion correctionMeasuring lens qualityThe Modulation Transfer Function (MTF)Measuring MTF in practiceSpot diagrams and the PSFMTF through focus and depth of focusField-dependent MTF: sagittal vs tangentialColor, geometry, and illumination measurementsScalar quality summariesLens optimizationThe high-level idea: design as optimizationThe forward model: ray-tracing and spot diagramsFrom hand calculation to softwareTradeoffs: the design is always a compromiseTolerancing: from the nominal design to a manufacturable oneA short bestiary of classic designsThe lens as a system: cardinal points, pupils, f-number, T-stopScaling laws in opticsLohmann's scaling laws: why a good lens is heavyThe gigapixel barrier for a single lensThe escape: monocentric multiscale opticsCapture everything, crop later: the spatial cousin of the light fieldSpecial opticsTilt-shift and the Scheimpflug principleFisheye and non-rectilinear projectionMirrors: catadioptric and reflecting systemsPeriscope / folded-lens design (smartphone telephoto)Anamorphic opticsStereo (3D) lensesMacro, microscope objectives, and telescopesTeleconvertersFrom shaped glass to thin structures: Fresnel, diffractive, GRIN, metalensesFocusFocus mechanicsFocusing a compound lens: beyond unit focusingFocus actuatorsFocus stacking, macro, and focusing railsAutofocusContrast-detection AFPhase-detection AF (the split-pupil / stereo trick)On-sensor PDAF and dual-pixel AFDepth from focus / defocus, and learned subject AFWhere to focus: saliency, faces, and eyesFocusing in astrophotographyBokeh, focus stacking, and depth-of-field controlRecap: the geometry of focus (pointer, not re-derivation)The bokeh look: shape and structure of the blurExtending DoF: focus stackingControlling and faking DoFFake (synthetic) depth of fieldWhy phones must fake itWhere the depth (or subject) comes fromFrom depth to blur — the thin-lens circleRealistic bokeh — why a Gaussian looks fakeOcclusion-aware compositing and mattingFailure modesGlare suppressionWhere stray light comes from: flare, ghosting, veiling glareHardware suppressionComputational deflare and glare deconvolutionOptical stabilizationThe problem: hand-shake and the blur budgetOptical stabilization: lens-shift vs. sensor-shift (IBIS)Digital / electronic stabilization and the computational alternativesThe eye as an optical instrument: vision and its correctionThe eye as a cameraRefractive errors: the eye out of focusCorrecting visionPresbyopia and the bifocal problem: from bifocals to AF glassesMULTIPLE EXPOSURE IMAGINGDenoising by averagingWhy averaging works — the $1/\sqrt N$ derivationWhen the plain mean is wrong: robust combinationCalibration frames: what averaging can't fixHandheld low light: the phone in your pocketDeep-sky astrophotography: averaging at the extremeHDR mergingThe HDR challengeData capture: how to vary the exposureCurve calibrationCombining exposuresOptimizing the capture and mergeIn-sensor HDR: dynamic range without a bracketApplication to cell phones: HDR+ and burst imagingWhy a phone shoots a burst, and why it underexposesThe HDR+ pipeline: align and robust-merge in rawFrom burst HDR to burst super-resolutionMultiframe or burst super-resolutionThe reconstruction principle: many coarse grids make one fine gridSub-pixel registration is the whole gameFusion: from scattered samples to a sharp imageAccidental versus deliberate offsets: hand tremor and pixel-shift sensorsThe learned eraWhere it runs outManual panorama stitching from multiple viewsThe scenario, and the one rule: rotate, don't translateRefresher: pinhole projection is "divide by depth"Why you don't need 3D: depth cancels for a pure rotationHomographies and homogeneous coordinatesSolving for $H$ from correspondencesWarping and assembling the panoramaAnother application: document flattening and mergingAutomatic panorama stitching from multiple views and feature matchingWhy not brute force, and the two sub-problemsThe feature pipeline, recalled from Part 7RANSAC for a homographyBlendingWhy a hard seam is visible — the photometric mismatchFeathering / alpha blending — and why it ghostsTwo-scale blending — the simple split (the pset method)Multiband / Laplacian-pyramid blending — a transition per bandPoisson / gradient-domain blending — paste gradients, solve for valuesSeam optimization — route the seam instead of fading itThe complete pipeline, end to endBells and whistlesOther projectionsBundle adjustmentMovement and parallax handlingContinuous panoramas (e.g. on cell phones)Incremental registration of a video streamMosaicking a moving strip (and why a central strip)Rolling shutter and exposure driftFocal stacks and depth of field extensionWhy limited depth of field is the problemA simple algorithm: sharpness = local high-frequency energy, then argmaxThe more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)Capturing the stack — hardware, and the magnification trapHyperspectral imaging, color wheelsWhy three numbers aren't enough — RGB as a 3-sample projectionBuilding the spectral stack — filter wheels, tunable filters, pushbroom, snapshotWhat it's for — material ID, agriculture, art and beyondPolarization imagingWhat a camera throws away — polarization as a third axis of lightCapturing the stack — rotate a polarizer, or a polarization mosaicCombining images at different polarizationsIntrinsic images with time lapseThe split, and why one image can't do itWeiss 2001 — the median of log-gradientsWhere this chapter belongs — passive vs. active illuminationLucky imaging (planetary / lunar astro)Atmospheric seeing — why one long exposure failsShoot thousands, keep the sharpestAlign and stack the survivorsA poor man's adaptive opticsMANY IMAGES AND PHOTO COLLECTIONSPhoto MosaicsThe tiling-and-matching pipelineColor correction and avoiding repeatsMulti-scale and irregular tilingsWhy it resolves into the target at a distanceRetrievalClassic CBIR — histograms and the text-retrieval analogyDeep retrieval — learned embeddings and CLIPRetrieval at scale — approximate nearest neighborMining what makes a place distinctiveAuto curationTechnical quality — the easy rejectsAesthetics — the hard, learned partGrouping, summary, and diversityArranging the collection, not just culling itLife logging camerasThe devices and how they fireThe memory-prosthesis reframingThe big-data problem passive capture createsPrivacy and ethicsInpainting Using Millions of PhotographsWhy self-similar inpainting isn't enoughScene completion from a huge databaseThe data is the prior — and its modern oppositePhoto tourismStructure-from-motion on internet collectionsFrom reconstruction to experienceLineagePhotobiosAlign and order the collectionLet the data fill the gapsCollection as experienceAverage ExplorerGalton's composite portraiture — and what it was forAlignment is everythingThe average of a categoryAn artistic lineageAverageExplorer: averaging made interactiveWhat an average is, and what it is good forPix 2 GPSGeolocation as retrieval over a geotagged corpusThe answer is a distributionMapping the collection itselfThe learned successorsPersonalized priorsPersonalized restorationPersonalizing generative modelsThe bargain and its ethicsArtistic projects with photo collectionsStatistical collage — SalavonAnticliché cameraPareidoliaDisplaying images togetherSelectionLayoutColor and coherenceThemesVIDEOMotion blur, temporal sampling, and resamplingA frame is an integral over time → motion blurTime is sampled → temporal aliasing, the wagon-wheel effectMotion blur is the temporal prefilter: the two are one tradeoffResampling in time: frame-rate conversionLagrangian vs Eulerian: the organizing distinction for the partVideo compression and motion compensationWhy video compresses far better than still × N: temporal redundancyMotion-compensated prediction: the core trickI, P, and B frames; GOP structureWhy this is "optical flow on a budget"Modern codecs in one breathVideo editingNon-linear editing: the timeline metaphorSummarization: keyframes, fast-forward, and highlightsFun temporal filters: reduce-over-timeTranscript-based editingIn-betweening: synthesizing the frames an edit needsCoda: storyboards, interviews, and where this part landsFrame interpolation and slow-motion synthesisWhy interpolate: faking slow-motion and up-converting frame rateInterpolation = morphing between adjacent framesFlow-based interpolation: warp both frames to the midpoint and blendLearned synthesis: Super SloMo and FILMHybrid low/high resolution and frame-rate imagingThe space–time bandwidth trade-offThe hybrid two-camera architectureMotion from the fast streamApplications: deblur, space–time super-resolution, video from stillsModern descendantsVideo stabilization and rolling-shutter correctionWhat stabilization is: a camera-path signal to be smoothedStage 1: estimating the camera trajectoryStage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)Stage 3: re-rendering and the stabilization↔crop tradeoffRolling-shutter correction: per-row pose and rectificationTime-lapse photographyCapturing a time-lapse: interval, shutter, and day-to-night transitionsDeflickering: stabilizing exposure, white balance, and lightHyperlapse: stabilizing a time-lapse that also movesMining time-lapses from the internetFactoring a time-lapse: reflectance, illumination, and intrinsic imagesVideo texturesFinding good transitionsPlaying itRelatives and descendantsLIGHT FIELDS AND PLENOPTIC CAMERASLight fields 101Capture rays, not pixelsThe plenoptic function, reduced to four dimensionsRay and point are dualReading the 4-D structure through its 2-D slicesRendering a new view by looking up raysLight fields vs. plenoptic and radianceLight field camerasIntegral photography: Lippmann's fly's-eye plate (1908)The plenoptic camera: a microlens array on the sensorLytro: the consumer plenoptic cameraCommercial light-field cameras beyond LytroCamera arrays: a grid of full camerasThe spatial↔angular tradeoffTwo strategies, and why arrays and microlenses are dualOther light field acquisition setupsOne camera on a gantry: sample the aperture in timeHandheld, unstructured capture: let the poses be irregularCatadioptric capture: one sensor, many viewpoints at onceCoded aperture in time: sweep the pupil itselfThe everything-else, mappedRefocusing and synthetic apertureReconstructing a photo: it is all about which rays you sumDigital refocusing is shift-and-addA focal stack from one capture, and an all-in-focus imageFourier-slice photography: the fast versionReading refocus and depth off an epipolar sliceSynthetic aperture: an aperture the size of a roomAberration correction in light fieldsAn aberration is misrouted raysRe-routing each ray to the ideal-lens positionThe trade: ray bookkeeping instead of glass, and only what you sampledLight field aliasing and 4D Fourier analysisThe light field is a sampled signal, and its samples are viewpointsDepth is slope is spectral orientationThe bowtie: a spectrum shaped by the scene's depth rangeHow densely must you sample? The plenoptic-sampling boundGeometry buys back samples: the depth-vs-views tradeoffUnder-sampling looks like a ghostWhere this sitsLumigraph and shape priors for sharper light field renderingPure light-field rendering blurs because it has no shapeThe Lumigraph: reproject onto a geometry proxy, then blendUnstructured inputs: free-hand views, no grid requiredSurface light fieldsLight field microscopyThe optical setup: a microlens array at the intermediate image planeThe spatial-versus-angular trade: a coarse 3-D volumeRecovering the volume: synthetic refocusing, then 3-D deconvolutionWhy single-shot 3-D is the whole pointLight field networksNeRF: a radiance field rendered by volume integrationLight field networks: a ray straight to color, in one evaluationA family of neural light fieldsTest-time training: the network as the per-scene priorHand-off: from rays to radiance fields and generationPractical aspects of light field camerasDo you lose all that resolution?Can you do video? Is it practical? The bandwidth problemThe time dimension is high speedAdjacent frontiers, brieflySo, do I get my camera?MULTI-APERTURE IMAGINGCamera arrays: one rig, many instrumentsOne rig, four instrumentsCommercial arrays: the Light L16Where this sitsBullet timeMulti-camera phonesCOMPUTATIONAL OPTICS AND CODED IMAGINGWavefront codingWhy you cannot just deblur defocusThe fix: re-engineer the blurThe cubic phase plateThe depth-invariant PSF and a single deconvolutionWhat it costsWhere it sitsCompressive sensingSub-Nyquist: fewer measurements than unknownsTwo ingredients: incoherent measurements and sparsityRecovery by $\ell_1$: the geometry of basis pursuitWhy it works: the restricted isometry property, intuitivelyThe single-pixel cameraWhere compressive sensing pays off — and where it does notCoded apertureThe bad forward operator of a clear apertureA mask designed for a flat, zero-free spectrumOne shot, two outputs: depth and an all-in-focus imageThe design criterion: which pattern?Coded-aperture pairs: splitting the trade across two shotsHeterodyning the light field: dappled photographyWhere it sitsPhase-coded aperturesA reminder on wavefront codingFocus sweepThe lattice-focal lensWhere it sitsCode in time (phase, amplitude)Why ordinary motion blur is (almost) unrecoverableThe flutter shutter: chop the exposure into a codeThe decode: one deconvolution, a sharp moving objectAmplitude in time, and phase in timeCoded strobing, temporal multiplexing, and compressive videoMotion-invariant photographyWhere it sitsTheoretical analysis of imaging systems in the 4D light field Fourier domainThe light field's spectrum, and the one move that explains everythingEvery camera is a different slicePutting cameras on one footing: the Bayesian comparisonThe lattice-focal lens: tiling the wedgeThe upper bound — and the gap we have not closedFrom cameras to light transport: the same spectral lensWhere this leaves the partEnd-to-end optimizationThe pipeline as one differentiable graphBackpropagating into the glassWhat gets designed: a height map, not a hyperparameterA gallery of deep-optics resultsWhat it costs, and where it can go wrongWhere it sitsFourier opticsLight as a wave: amplitude, phase, and what "coherent" meansDiffraction is a Fourier transformA lens computes a Fourier transformThe pupil function is the transfer functionThe diffraction limitAberrations are pupil phaseEvery code in this part is a choice of pupilFourier ptychography: synthesizing a bigger pupilExotic / advanced opticsLensless imagingGRINMetalenses and advanced crazy optics à la Barbastathis (coherent though)Non-linear opticsOptical modulators (spatial light modulators): DMD and LCD/LCoSCOMPUTATIONAL SENSORSAssorted pixelsDual-pixel and phase-detect pixels: buying depth and focusClear, white, and other color-filter variantsPolarization pixelsSpatially varying exposure: assorting for dynamic rangeThe common threadModern sensors (quad Bayer, in-sensor HDR, and beyond)Quad Bayer, Tetracell, and nona-binningIn-sensor HDR: capturing range before the mergeDual-pixel autofocus, on the same sensorBSI and stacked sensors: compute under the pixelsThe global-versus-rolling shutter tradeBeyond: nano-prism, organic, and event pixelsOn-sensor HDRStaggered / multiple-exposure readout (DOL-HDR)Dual (and triple) conversion gain (DCG)Split-pixel: a large and a small photodiodeSpatially-varying exposure (SVE) / assorted exposuresLateral overflow integration capacitor (LOFIC)Logarithmic, self-resetting, and counting pixelsOn-sensor HDR versus multi-frame HDRDepth sensorsStereo: passive triangulationStructured light: projecting the textureLiDAR and direct time-of-flightTime of flightPassive depth from one camera, in passingThe menu, in one viewSingle-photon sensors (SPAD, avalanche, photon counting)From avalanche gain to a single-photon clickPhoton-counting arrays: zero read noise, shot-noise-limitedThe Quanta Image Sensor: a different road to one photonWhat it costs: dark counts, dead time, fill factor, data rateDoppler / velocity imagingThe Doppler shift, as a velocity sensorThe instruments: vibrometry, Doppler LiDAR, radarWhere it fitsEvent sensorsHow it works: per-pixel change detectionThe upside: microseconds, dynamic range, no blur, little dataThe downside: no picture, and a new kind of dataUses, lineage, and the contrast with single-photonSpecialized and research sensorsGeiger-mode avalanche-photodiode arrays: photon-counting laser radarDigital-pixel focal-plane arraysScientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaicsStacked and processing-in-pixel sensorsCurved focal planesBeyond the visible, and filter-array sensorsRadiation-hardened and defense focal planesThe chapter's pointExtra sensors and non-visual dataAccelerometer and gyroscope: the inertial measurement unitSound: microphones, audio-visual sync, and the visual microphoneGPS: geotagging and placeCompass and magnetometer: heading and orientationNear-infrared: the cut filter, dark flash, and NIR-assisted denoiseTemperature: dark-current compensationUltra High speed ImagingStreak cameras: sweeping time onto a spatial axisFemto-photography: a movie of light in flightCompressive ultrafast photography: a single-shot coded streakTransient imaging and looking around cornersThe bridge to direct time-of-flight and LiDARCOMPUTATIONAL ILLUMINATIONFlash photographyFlash / no flashRamesh's multiflashRemoving flash artifactsDark flash (plus Stasi version!)High-speed and stroboscopic photographyFreezing motion: the microsecond strobeThe trigger problemStroboscopic multiplicity: a sequence on one frameDigital descendants: LED strobes and high-speed camerasIllumination-based mattingThe well-posed case to beat: chroma keyThe magic prism: Disney's sodium-vapor processNear-infrared and time-multiplexed mattingFlash/no-flash matting: separation by falloffThe throughline: control the capture, not the priorSeparation of Direct and Global IlluminationThe frequency insightNayar's program: programmable, structured illuminationToward coherent separationLight domesThe reflectance field and one-light-at-a-time captureRelighting by linear combinationScaling down: tabletop LED domesScaling out: the dome taken into the wildThe found dome: the eyeAutomatic aesthetic lightingComputational bounce flashDrone lighting: flying the light into placeDual photographyLight transport as a matrixHelmholtz reciprocity: transpose the matrixThe unsettling reach: privacy and seeing the unseenCoherent imagingThe confocal principle: rejecting out-of-focus lightSeeing through scattering mediaThe part in one line3D AND DEPTHMultiple view geometryTwo views: stereo and disparityTwo-view geometry: epipolar lines, and the essential and fundamental matricesWhat "depth" means, and where it comes fromDepth is the z-coordinate, not the ray lengthWhere depth comes from: a cue-and-sensor inventoryRelative vs metric: the scale you usually don't haveMonocular depth estimation (one image → depth)Why one image cannot determine depthRelative depth, and the scale-and-shift ambiguityFrom hand-built priors to borrowed diffusion priorsWhat the maps are good forSingle-image 3-D: tour into the picture, photo pop-up, 3-D Ken BurnsThe universal recipe, and why holes are the hard partTour Into the Picture: the spidery meshAutomatic Photo Pop-up3-D Ken Burns and 3-D photos: the monocular formMulti-view 3-D reconstruction: the classic pipelineThe pipeline, stage by stageWhy it is brittle: and why SfM survives anywayStructured light scanningProjector as inverse camera: triangulation with trivial correspondenceThe coding ladder, and the frames-versus-motion tradeCalibration and failure modesPhotos → radiance fields and Gaussian splatting (NeRF, 3DGS)Two goals, one diagramInverse differentiable rendering, and the discontinuity that forced fuzzinessNeRF: a scene as a tiny neural networkDo we even need the network?3-D Gaussian Splatting: the lessons without the deep learningThe practical recipe: and what is baked inRelaxing the assumptions: NeRF in the wildFeed-forward (amortized) 3-D: skip the per-scene optimizationAmortization: pay once, reuse foreverThe line: DUSt3R, MASt3R, VGGTThe punchline, and the loop it closesThe trade, and the data dependencyRe-photographyWhy you cannot just overlayThe real-time guidance loopWhere it sitsThe landscape, and is 3-D a "fake task"?The field as a landscape, not a lineIs 3-D a "fake task"?The counterpoint, and the frontierINTEGRAL AND IMMERSIVE IMAGINGStereo glassesWheatstone's stereoscope (1838)Routing a different image to each eyeShooting and synthesizing a stereo pairVR gogglesThe three levelsThe optics: a microdisplay and a magnifier per eyeTracking, latency, and why headsets used to make people sickPassthrough, mixed reality, and where today's products sit3D displays with accommodationFour ways to deliver a focus cueThe other gaps, and the ultimate displayLenticular displaysLippmann's integral photography: the common ancestorMulti-view, and the resolution–views tradeoffLight-field telepresence: Google StarlineDisplay depth of field and antialiasingHolographyLippmann and Gabor: recording the waveOff-axis holography and the space-bandwidth wallComputational holographyRetinal projectionThe Maxwellian view: focus set by the display, not the eyeFrom the virtual retinal display to laser eyewearThe frontier: writing to individual conesREVEALING THE INVISIBLEAccidental camerasThe accidental pinhole: a window is a cameraThe accidental pinspeck: the anti-pinholeThe occluder as a crude lens, and recovery as deconvolutionCorners and doorways: an edge that resolves the hidden roomWhere else the world hides a cameraReflections in the eyeThe cornea as a catadioptric mirrorThe geometry: from a corneal pixel to a direction in the worldWhat the recovered reflection is good forEyes for relightingMotion and video magnificationThe Lagrangian precursor: track, then exaggerateEulerian video magnification: amplify the time series at each pixelWhy amplifying brightness amplifies motionPhase-based magnification: move the motion into phaseWhat it reveals: vital signs, structures, materials, modesVisual microphoneFrom sub-pixel motion to a sound waveformBandwidth: high-speed cameras and the rolling-shutter trickHow good is the copy? The object's frequency responseThe active cousins: laser vibrometry and interferometryCorner cameraThe edge as a one-dimensional apertureFrom a faint gradient to a usable signalWhat the corner can and cannot tell youActive non-line-of-sightThird-bounce geometry and time-of-flightThe hardware: photographing light in flightFrom back-projection to fast, exact inversionThe trade, stated plainlyPassive non-line-of-sightThe occluder is what makes it solvableA deconvolution where the lens is unknownOne dimension: the corner cameraTwo dimensions from a single photo: computational periscopyActive versus passive, the ledgerMm-wave, wifiWhy radio walks through wallsTime of flight, again — radar is NLOS with a longer waveFrom a radio smear to a human skeleton: the learned mapWhat it sees, and what it costs usADJACENT FIELDS AND APPLICATIONSOptical computingAstroExtreme long exposureTracking, stacking, and selection (pointers)X-rayMedicalMicroscopyMm-waveMusic, soundFluorescenceOpto-acousticUltrasoundAerial imagingComputer visionRobotics, drivingHUMAN FACTORSHuman factors and the art of photographyMake better photosTypical shooting scenariosMacro photographySpecial effect photographyFun artsy stuffPerception of artEthics of computational photographyComputational models of perceptionSpatial (and spatio-temporal) visionUser studiesAccessibility: photography by and for blind usersBlind camera — capture without a sighted operatorThe social and personal practice of photographyIMAGE FORENSICS AND AUTHENTICATIONImage ForensicsThe problem and the threat modelSensor and pipeline traces: PRNU, CFA, and noiseCompression, geometry, and metadata forensicsDeepfakes, GAN/diffusion fingerprints, and learned detectionWhy forensics is evidence, not proof — and the hand-off to provenanceAuthentication and Provenance (C2PA)From detection to attestation: trustworthy cameras and watermarkingC2PA and Content Credentials: the standardAI disclosure, watermarking, and the regulatory pushLimits, critiques, and the forensics partnershipSYSTEMSProgrammable and modular camerasImage processing librariesLightroom-style raw developersPhotoshop-style editorsNetworking and image transportPhotography programming on phonesPERFORMANCE ENGINEERING AND HALIDE8-bit and fixed-point arithmeticInteger versus fixed-point: where the binary point sitsRounding, dithering, and the banding trapSaturation and the width of the accumulatorfp16 versus bf16: the exponent–mantissa bargainint8 quantization for neural inferenceWhen float is non-negotiableAlgorithmic speedupsSeparability — a 2-D pass for the price of two 1-D passesRecursive / IIR filters — a running state, cost independent of radiusIntegral images / summed-area tables — any box sum in four lookupsFast median filters — a sliding histogram for a constant-time medianPyramids and multiscale — do the large-scale work on small imagesDiscretize the range to accelerate non-linear filtersDownsampling and edge-aware upsamplingWhere this goes nextAutomatic search for fast methodsThe problem is a curve, not a pointWhy search beats a hand-shrunk CNNMa et al. 2022: searching structure and parameters togetherThree axes of "search instead of design"The broader family, and what it costsWhere this goes nextModern CPUs: memory hierarchy, parallelism, and what it takes to go fastWhy moving data, not doing math, is the bottleneckThe memory hierarchy and localityThe roofline: is my kernel compute- or memory-bound?The forms of parallelismWhat it takes to leverage a modern machineWhy photography is hard for the machineWhere this goes nextHardware backends: GPU, NPU, DSPGPU — the data-parallel workhorseThe neural accelerator: NPU and TPUDSP — the real-time control loopISP, FPGA, and ASIC: the fixed-function endThe heterogeneous SoC, and the spectrum to carry awayOn-device ML runtimes, and why the work stays on the phoneHalide: Decoupling Algorithms from SchedulesWhat it means to separate the algorithm from the scheduleWhy image pipelines are uniquely hard to optimizeThe split was always there: done by handThe scheduling spaceAuto-scheduling — letting the compiler searchResults, impact, and reachGradient Halide — differentiating the pipelineWhere this goes nextHalide programmingThe three nouns: Func, Var, ExprA first image pipeline: brighten, then blurReductions: RDom, sums, and histogramsThe default schedule, and seeing the loopsScheduling the loops within a stage: reorder, split, tile, vectorize, unroll, parallel, fuseThe heart of it: producer–consumer granularity (compute_at, store_at)The schedule ladder, with numbers: ten times faster from one lineBoundaries and boundsSame algorithm, new machine: the GPULetting the compiler schedule: the auto-schedulerThe development loop: correctness, measurement, and benchmarking hygieneEfficient neural network inferenceQuantization: fewer bits per weightPruning: fewer weightsKnowledge distillation: a small student, a big teacherLow-rank and tensor factorizationEfficient architectures: cheapness designed inNeural architecture search: let the machine design itHardware-aware deployment and the rooflineWhere this leaves the partCONCLUSIONS, DISCUSSIONRecap in contextModern phones, multiple apertures, pano, HDR+Recap: a modern mirrorless cameraRecap: a modern cell phone multi cameraLightroomPhotoshopWhy phones are so good (at photography)Computation beats glassThe whole pipeline is co-designedMachine learning and data at scaleMany small cameras for one big oneThe human and system advantagesThe hard caveats: physics still wins where it mustThe throughlineBACK MATTERBibliographyGlossaryAcronymsTerm indexAPPENDICESRefreshersLinear algebraCalculus: derivatives, gradients, integralsOptimization and regressionProbability and information theoryMachine learning and deep learningProgramming: Python, C++, and PyTorchProblem Set 0 — Environment and C++ basicsSummaryInstallation and Environment SetupC++SubmissionProblem Set 1 — Image class, point operations, and colorSummaryThe Image ClassBrightness and ContrastMore Image Class MethodsColorspacesSpanish Castle IllusionWhite BalanceProblem Set 2 — Convolution and the bilateral filterSummarySmart AccessorBlurringDenoising using Bilateral FilteringExtra creditSubmissionProblem Set 3 — Denoising and demosaickingSummaryDenoising from a sequence of imagesDemosaicingEdge-based greenRed and blue based on green6.865 only (or 5% Extra Credit): Sergey Prokudin-GorskyExtra credit (maximum of 10%)Problem Set 4 — High dynamic rangeSummaryHDR mergingTone mappingExtra credit (10% max)Problem Set 5 — Resampling, warping, and morphingSummaryResamplingWarping and morphingExtra creditProblem Set 6 — Homographies and manual panoramasSummaryClass MorphHomogeneous CoordinatesLinear AlgebraWarp and Image with a HomographyCompute Homography from 4 Pairs of PointsBounding boxesExtra Credit (up to 10% total)Problem Set 7 — Automatic panoramasSummaryPrevious Problem Set CodeClass MorphHarris Corner DetectionDescriptor and correspondencesRANSACAutomatic panorama stitchingBlendingMini planet6.8370: Stitch N Images (6.8371: Extra Credit 5%)Make your own panoramaExtra credits (10% max)Problem Set 8 — Non-photorealistic renderingSummaryPaintbrush splattingPainterly renderingOriented painterly renderingYour imagePaper Review (6.865 only)Extra creditsProblem Set 9 — Make-your-own, video, and ethicsSummaryMake Your Own AssignmentEthical issues in computational photographyAssignment ListsEXIF and image metadataWhat EXIF isThe fields, grouped by what they describeHow far to trust itPrivacy: the metadata that follows the pictureReading and writing EXIFDNG: the Digital NegativeWhat DNG is, and the problem it solvesInside the containerWhat the raw payload looks like: mosaic vs linearThe color recipe: matrices, profiles, and white balanceOpcodes: corrections the decoder must applyCompression, and embedding the originalWhere you meet DNG: adoption and relativesTrade-offs, and DNG's relation to EXIFRendering a raw DNGThe pipeline, step by stepWhere Lightroom's "look" actually comes fromWhat we would ask an Adobe engineer to checkTwo kinds of DNG, and the special casesEasy mistakes (most of which we made)DatasetsClassification and featuresSuper-resolutionDeblurring and restorationDenoisingHDR and tone mappingRetouching and enhancementDepth and motionLight fieldsColor and white balanceFacesInpainting, segmentation, and mattingImage qualityA camera-feature wish listExposure, ISO, and dynamic rangeBracketing more than exposureFocus and depth of fieldComputational raw and the sensorMotion data, metadata, and workflowPanorama and multi-shotThe interface and the ecosystemHow this book was createdTwo documents, not oneCompiling a sectionFigures as codeGenerative imagery: cover art and 3DVerification and reviewKeeping a long book coherentThe toolchainWhat the machine did, and what it did notWho wrote what: a per-part estimateThe course tutor: a local, book-grounded AI teaching assistantWhat it is, and what it is forLocal-firstGrounded in the book: retrieval-augmented generationIt links, it shows equations, it shows figuresTwo front-ends, one coreWhat the instructor seesPrivacy and candorThe semi-automatic grading systemThe shape (to be confirmed)Automatic versus human (to be confirmed)To be filled in (from the instructor)Under the hood: prompts, patterns, and verifiersPrompt patterns that made it workThe verifier suiteWhy this is the interesting partReading a Lytro light fieldThe container: the LFP/LFR formatThe pipeline, step by stepFrom hexagonal lenslets to a uniform gridWhy a white imageOpen questions and where we approximateEasy mistakes (most of which we made)File conversion toolsJPG → PNG, without an alpha channelDNG → a linear PNGA Lytro capture → a Stanford-style light-field archiveHow they run in the browserEasy mistakesThe interactive figures — a prompt-by-prompt making-ofSummaryLorentz resonance — fig-lorentz-resonance (Figure 2.1.8)Rainbow droplet — fig-rainbow-droplet (Figure 2.1.14)Diffraction wave simulation — fig-diffraction-wave-sim (Figure 2.1.29)Exposure-triangle simulator — fig-exposure-triangle-sim (Figure 2.11.5)Repeated quantization — fig-repeated-quantization (Figure 3.1.4)JPEG generation loss — fig-jpeg-generation-loss (Figure 3.17.9)Exposure round-trip (JPEG) — fig-exposure-jpeg-roundtrip (Figure 3.17.10)1-D sampling pipeline — fig-sampling-1d-demo (Figure 3.10.4)2-D sampling pipeline — fig-sampling-2d-pipeline (Figure 3.10.6)Mitchell–Netravali bicubic (B,C) — fig-bicubic-bc (Figure 3.11.13)Rotation resample challenge — fig-rotation-resample-interactive (Figure 3.11.18)Mini-Lightroom — fig-mini-lightroom (Figure 3.18.4)Poisson blending — fig-poisson-blend (Figure 5.1.5)Bilateral grid (3-D) — fig-bilateral-grid-3d (Figure 5.2.15)Perspective montage — fig-perspective-montage (Figure 6.6.13)Beier–Neely morph — fig-beier-neely-demo (Figure 6.3.4)Live face landmarks — fig-face-landmarks-live (Figure 7.10.2)Lens optimizer — fig-lens-optimizer-demo (Figure 9.4.4)Full auto-panorama — fig-pano-stitch (Figure 10.7.9)CLIP-IQA curation — fig-clip-iqa (Figure 11.3.2)Photobio time-lapse — fig-photobio-demo (Figure 11.7.2)Refocus shift geometry — fig-refocus-shift-geometry (Figure 13.4.2)Chroma key — fig-chroma-key (Figure 8.11.7)Portrait-lighting simulator — fig-portrait-lighting-sim (Figure 2.11.26)Aberration explorer — fig-aberration-explorer (Figure 9.1.6)Interactive demo indexSharing, linking, and embedding a demoIntroductionFundamentals — light, optics, sensors, colorBasic image processing and the ISPComputational tools — machine learning and diffusionEdges matter — gradient-domain and edge-preservingWarping and morphingMatching pixels across space and timeSingle-image computational photographyOptics, lenses, and aberration correctionMultiple-exposure imaging — HDR and panoramasMany images and photo collectionsVideoLight fields and plenoptic cameras3-D and depthAppendices and end matterNot yet placedBACK MATTER
Glossary Acronyms Index 💬 Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.×
Computational Photography, an AI-powered Slopendium
« Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte. »
“If I had more time, I would have written you a shorter letter.”
Blaise Pascal
30 parts · 288 chapters · 252 drafted · blue = drafted, links open the chapter; grey = outline only
This is a very preliminary draft, and even the organization is still in flux. Feedback is welcome on content and organization, but fine-grained critique (wording, typography, line-level polish) is probably premature.
To leave feedback, select any text on a page and click the note / highlight button that pops up, or open the annotation panel with the tab at the top-right (‹). Notes are visible only inside our private review group.
Preliminary draft · v0.1.226
show Parts Chapters Sections collapse all open in Chapter pages Section pages
1.1 From Digital to Computational Photography 1.1.1 Computation is the new optics 1.1.2 The goals of computational photography 1.1.3 What makes a technique useful 1.1.4 A field in transition: from hand designed algorithms to learning to generative AI 1.1.5 Does photography still matter in the age of generative AI? 1.1.6 A field at a crossroads 1.1.7 It's not just about photography 1.2 How to read this book 1.2.1 Machine-readable by design 1.3 How this book was made 1.4 What programming language for computational photography? 1.4.1 On the desktop: Python, C++, and Halide 1.4.2 In the browser: JavaScript and the web stack 1.4.3 On the phone: Android and iOS 1.4.4 Libraries 1.4.5 Vibe coding 1.5 Problem sets 2.1 Light and physics 2.1.1 Light: rays, waves, and the spectrum 2.1.2 How light is created 2.1.3 Light power and brightness 2.1.4 Summing coherent vs. incoherent light 2.1.5 Reflection, refraction, and what happens at a surface 2.1.6 Polarization 2.1.7 The color of objects: illumination times reflectance 2.1.8 The BRDF and the look of materials 2.1.9 Illuminants 2.1.10 Radiometry: radiance, irradiance, exposure, and falloff 2.1.11 Global illumination 2.1.12 Wave effects, diffraction, and the diffraction limit 2.2 Perceptual color and trichromatic vision 2.2.1 Visual system anatomy: 2.2.2 Color 2.2.3 Opponent process and the multistage model 2.2.4 So what are the primary colors? 2.3 Human Vision 2.3.1 Light adaptation 2.3.2 Lightness constancy 2.3.3 Color constancy 2.3.4 Contrast 2.3.5 Spatial vision 2.3.6 Temporal vision 2.3.7 Attention and eye movements 2.3.8 The visual system makes things up 2.4 Animal eyes 2.4.1 The optics: many ways to form an image 2.4.2 The color vision: opsins remixed 2.5 Measuring and encoding color 2.5.1 analysis vs synthesis and non-orthogonality 2.5.2 Measuring color 2.5.3 Chromaticity diagram 2.5.4 Linear vs Gamma vs. log encoding 2.5.5 RGB color spaces 2.5.6 Non-linear perceptually-uniform color space 2.5.7 HSV, HSL, and cylindrical color spaces 2.5.8 Reproducing color 2.5.9 Skin tones 2.6 Pinhole image formation and linear perspective 2.6.1 Pinhole imaging and the perspective projection 2.6.2 Homogeneous coordinates 2.6.3 Camera in a general configuration 2.6.4 Intrinsics, extrinsics, and what cropping really does 2.6.5 What perspective preserves, and what it destroys 2.6.6 Wide-angle distortion: spheres bulge and faces stretch at the edges 2.6.7 Photography with focal length: framing, magnification, and compression 2.6.8 Depth, ray length, and unprojection 2.7 Lens image formation 2.7.1 Single lenses: refraction and Snell's law 2.7.2 Thin lens optics 2.7.3 Aperture and the f-number 2.8 Image measurements as integrals 2.8.1 The plenoptic function 2.8.2 The pixel integral 2.8.3 Exposure values (EV) and stops 2.8.4 Splitting the integral 2.9 Depth of field 2.9.1 The circle of confusion 2.9.2 Depth of field versus depth of focus 2.9.3 Choosing the maximum circle of confusion 2.9.4 The double cone, and the near and far limits 2.9.5 Hyperfocal distance 2.9.6 The surprising invariance 2.9.7 Background blur beyond the focus plane 2.9.8 Defocus is not a blur of the image 2.9.9 Sensor size scales depth of field 2.10 Motion blur 2.10.1 Blur from subject motion 2.10.2 Blur from camera shake 2.10.3 How shift-invariant is camera-shake blur? 2.10.4 Which dominates: translation or rotation? 2.10.5 The hand-holding rule of thumb 2.10.6 Shake from the shutter and mirror 2.10.7 Panning: fighting subject motion by moving the camera 2.10.8 Freezing motion with light 2.11 Sensors: photosites, CCD vs CMOS 2.11.1 The photoelectric effect 2.11.2 Photon to number: the photosite 2.11.3 Spectral sensitivity: matching the eye, catching photons, and white balance 2.11.4 Beyond the visible spectrum: near-infrared, thermal, and ultraviolet 2.11.5 Analog-to-digital conversion 2.11.6 CCD versus CMOS 2.11.7 Time integration: the exposure 2.11.8 Shutters 2.11.9 A taste of modern sensor tricks 2.11.10 The impact of sensor size 2.11.11 How far sensors have come 2.12 Sensing color: multiplexing strategies 2.12.1 Multiplexing in time 2.12.2 Multiplexing in space: the color filter array 2.12.3 Multiplexing across separate sensors: the beam-splitter 2.12.4 Multiplexing per pixel by dispersion: color routers and nano-prisms 2.12.5 Multiplexing per pixel by tunable absorbers: quantum dots 2.12.6 Multiplexing in depth: stacked photodiodes 2.12.7 Hybrid strategies 2.12.8 Beyond trichromatic capture: full spectrum and multispectral 2.13 Noise, signal-to-noise ratio and dynamic range 2.13.1 Noise sources 2.13.2 Noise variance is affine in pixel brightness 2.13.3 The algebra of noise 2.13.4 Signal-to-noise ratio: it's about ratios 2.13.5 Dynamic range 2.13.6 Regimes, and how close we are to the limits 2.14 Imaging as a linear system 2.14.1 Linear systems from a billboard to an image 2.14.2 Linear systems from ray space (light fields) to an image 2.14.3 Space–time: imaging across time 2.14.4 When linearity breaks down 2.14.5 Where this is going: invertibility 2.15 Imaging as an inverse problem 2.15.1 Linear algebra: how hard is this problem? 2.15.2 Priors and the manifold of natural images 2.15.3 Probability: the Bayesian view 2.15.4 Optimization, inference, loss 2.15.5 When you can design the operator — and a teaser of information theory 2.15.6 Recap: what makes imaging hard, and where it's going 2.15.7 Harder inverse problems: factorization 2.16 Limitations of the medium 2.16.1 Compensation, accentuation, conflict 2.16.2 The picture is flat: depth and its cues 2.16.3 The picture is static: time and motion 2.16.4 One viewpoint, and a finite frame 2.16.5 The contrast is limited: dynamic range and gamut 2.16.6 Resolution: the limitation we are overcoming 2.17 Displays 2.17.1 The range of displays 2.17.2 Display technology 2.17.3 HDR displays and dual modulation 2.17.4 The film movie projector 2.17.5 Printing 2.17.6 Why the display characteristics matter 2.17.7 Distance, resolution and acuity 2.17.8 Sharpening for the display: size and distance 2.17.9 Light level viewing conditions 2.17.10 Robustness of perspective to the viewer's viewpoint 2.17.11 Gamut and gamut mapping 2.17.12 Color management, ICC, and industry standards 3.1 Exposure 3.1.1 Basic photography: exposure settings — shutter, aperture, and ISO 3.1.2 Metering: from printed guides to the sensor 3.1.3 f-stops versus T-stops 3.1.4 From exposure to photometry 3.1.5 Exposure modes, UI, and auto-ISO 3.2 Lenses 3.2.1 Lenses and focal length 3.2.2 Aperture and the diaphragm 3.2.3 Focusing the lens 3.2.4 Image stabilization 3.2.5 Keeping the lens clean 3.2.6 Lens filters: polarizers, ND, and graduated ND 3.3 Focus, autofocus, and depth of field 3.3.1 Manual-focus aids: making the focal plane visible 3.4 Cameras 3.4.1 UI and display 3.4.2 How the viewfinder works 3.4.3 Anatomy of a modern full-frame interchangeable-lens camera 3.4.4 Anatomy of cell-phone cameras 3.4.5 Camera versus phone 3.4.6 Types of cameras 3.4.7 Cameras beyond photography 3.4.8 A camera's other sensors 3.5 Video 3.5.1 Cinema lenses 3.6 Illumination and the flash 3.6.1 Natural illumination 3.6.2 Flash 3.6.3 Flash metering 3.6.4 Indirect illumination and bounce flash 3.6.5 Multiple-point lighting 3.7 Traditional and Digital Darkroom 3.7.1 The traditional darkroom 3.7.2 The digital darkroom 3.8 Programmability, or the lack thereof 3.9 Photographs are usually not passive objective recordings 3.9.1 Faithful is not the same as realistic 3.9.2 Choosing the instant, choosing the viewpoint 3.9.3 Speed, aperture, and focus: the photographer rules 3.9.4 Filters and lighting: building the scene's light 3.9.5 The darkroom: tone, dodging, and burning 3.9.6 Make-up and retouching: authoring the subject 3.9.7 Getting closer — systematic capture, and the biases that remain 4.1 Image representation 4.1.1 An image is an array 4.1.2 Float vs 8-bit vs more bits 4.1.3 Stochastic Quantization and Dithering 4.1.4 How the array sits in memory 4.1.5 Pixel coordinate conventions 4.1.6 Alpha and extra channels 4.1.7 Video and more dimensions 4.1.8 Pixel lookups and boundaries 4.1.9 Beyond the pixels: basic metadata and EXIF 4.1.10 Three kinds of operation 4.2 What the numbers mean 4.2.1 What the numbers actually mean: encoding 4.2.2 What the numbers actually mean: color spaces 4.3 Developing, Testing and Debugging 4.3.1 The workflow: build, run, look at the picture 4.3.2 Principles 4.3.3 Test on inputs you can verify by hand 4.3.4 Finding good real test inputs 4.3.5 Per-algorithm debug recipes 4.3.6 Crashes and bounds 4.3.7 Vibe coding: writing image code with an LLM 4.4 Point operations 4.4.1 Three kinds of operation 4.4.2 The point-operation curve 4.4.3 Exposure: a multiply in linear light 4.4.4 Brightness vs. exposure 4.4.5 Contrast: steepen about a pivot 4.4.6 Black point, white point, and levels 4.4.7 The general tone curve 4.4.8 Basic color enhancement: saturation and vibrance 4.4.9 Converting to black and white 4.4.10 Lookup tables 4.4.11 Video color grading 4.4.12 Local point operations 4.5 Histograms 4.5.1 the histogram 4.5.2 the histogram depends on the encoding space 4.5.3 a histogram is a sampled estimate 4.5.4 histogram equalization 4.5.5 histogram matching 4.5.6 constraining the slope: Ward's histogram-based tone mapping 4.6 Global tone mapping 4.6.1 The dynamic-range problem 4.6.2 Two families: global and local 4.6.3 Global tone mapping: one curve for everything 4.6.4 Histogram equalization and Ward's bound 4.6.5 Local tone mapping and the halo problem: why naive local fails 4.6.6 Why log space pays off 4.6.7 The analog ancestor: the Zone System 4.6.8 HDR on every screen: the gain map 4.6.9 Where this is going 4.7 Beauty curves: the camera "look" 4.7.1 Why scene-linear looks dull 4.7.2 The shape of the curve 4.7.3 Highlights: roll off to white, not to a color 4.7.4 Saturation, and the rest of the "look" 4.7.5 The discipline 4.8 Neighborhood operations and convolution 4.8.1 Motivation: blur and sharpen 4.8.2 Convolution 101 4.8.3 The flip: where-from vs. where-to 4.8.4 Properties: the impulse, normalization, symmetry, commutativity 4.8.5 Nitty-gritty: finite images 4.8.6 A blur zoo 4.8.7 Separability 4.8.8 Gradient and oriented filters 4.8.9 Is imaging blur usually a shift-invariant convolution? 4.8.10 Where this goes next 4.9 Sharpening 4.9.1 Linear sharpening 4.9.2 Why it's called "unsharp mask" 4.9.3 Sharpen in gamma, linear, or log? 4.9.4 Non-linear sharpening 4.10 Fourier 4.10.1 Images as vectors in a high-dimensional space 4.10.2 why Fourier 4.10.3 Definition: one coefficient per wave 4.10.4 Sines are the eigenvectors of convolution 4.10.5 The 2-pixel example 4.10.6 Reading an image's Fourier transform 4.10.7 A small bestiary of transforms 4.10.8 The fine print: two limitations of Fourier 4.10.9 Windowing: the price of measuring a real spectrum 4.10.10 Application preview: can we deblur, given the blur? 4.11 Sampling, downsampling, and aliasing 4.11.1 Aliasing: a high frequency in disguise 4.11.2 Nyquist and the sampling theorem 4.11.3 In two dimensions, aliasing has a direction 4.11.4 Sampling in the frequency domain: spectral replicas 4.11.5 Seeing it: sampling and pre-filtering, hands on 4.11.6 The ideal reconstruction filter: sinc, and why it is unreachable 4.12 Resampling and upsampling 4.12.1 Domain operations: moving pixels around 4.12.2 Start with scaling up 4.12.3 The naive idea, and why it leaves gaps 4.12.4 Loop over the output, use the inverse transform 4.12.5 Nearest neighbor 4.12.6 Linear interpolation, in 1-D first 4.12.7 Bilinear, in 2-D 4.12.8 The convolution perspective 4.12.9 Better kernels: bicubic and Lanczos 4.12.10 Upsampling vs. downsampling: scale the kernel 4.12.11 Prefiltering when the transform isn't a clean scale 4.12.12 Beyond Nyquist-limited reconstruction 4.12.13 Where this goes next 4.13 Pyramids and wavelets 4.13.1 Halfway between space and frequency 4.13.2 The idea: process an image at many resolutions 4.13.3 The Gaussian pyramid 4.13.4 The Laplacian pyramid 4.13.5 Reconstruction: an exact encoder and decoder 4.13.6 Each level is a frequency band 4.13.7 What the bands look like on a real image 4.13.8 Candid limitations 4.13.9 Cousins: wavelets and steerable pyramids 4.13.10 Pyramid blending 4.13.11 Other applications 4.13.12 Multiresolution as a recurring theme 4.14 Image metrics 4.14.1 Full-reference vs. no-reference 4.14.2 why not just L2? 4.14.3 PSNR 4.14.4 SSIM 4.14.5 VDP and HDR-VDP 4.14.6 learned metrics 4.14.7 The right metric depends on the task 4.15 Denoising basics 4.15.1 what is noise? 4.15.2 denoising by averaging multiple shots 4.15.3 denoising from a single image 4.15.4 Spatial averaging and its limits 4.15.5 The bilateral filter: averaging by affinity 4.15.6 Denoising in the pyramid: coring 4.15.7 Denoise color more than brightness 4.15.8 noise estimation 4.15.9 The limits of denoising 4.16 Demosaicking 4.16.1 Reminder: the Bayer mosaic 4.16.2 The task: full RGB at every pixel 4.16.3 The naive approach: interpolate each channel on its own 4.16.4 Why naive interpolation zippers: averaging across an edge 4.16.5 Doing better: edge-directed interpolation 4.16.6 The harder half: red and blue, and color fringing 4.16.7 Green-based demosaicking: interpolate the color difference 4.16.8 Classic (non-learning) demosaicking: the general strategy 4.16.9 Related: the optical anti-aliasing filter 4.16.10 Quad-Bayer sensors: remosaic before demosaicking 4.16.11 Fuji X-Trans: a larger, irregular CFA 4.16.12 Beyond hand-tuned: joint denoising and learned demosaicking 4.16.13 Cross-reference: other ways to sense color 4.16.14 Where this sits in the pipeline 4.17 Auto-exposure and auto white balance 4.17.1 Auto-exposure: metering 4.17.2 white balance and color constancy 4.17.3 Automatic white balance 4.17.4 The limits of white balance, and CRI 4.18 File formats and compression 4.18.1 The big picture: none, lossless, lossy 4.18.2 Data versus metadata: EXIF 4.18.3 PNG: the format we read from 4.18.4 JPEG: compression by perception 4.18.5 RAW files: before the cooking 4.18.6 HDR formats: more than 8 bits 4.18.7 Modern formats 4.18.8 Other formats in passing: TIFF and GIF 4.19 Recap ISP, non-destructive editing: 4.19.1 A basic ISP 4.19.2 Pipeline design and tuning 4.19.3 The ISP, evolving: traditional → learned → generative 4.19.4 Recap 2: non-destructive editing 5 COMPUTATIONAL TOOLS
5.1 Linear Inverse Problems and Regression5.1.1 Blur is linear, so deblurring is inversion5.1.2 Images as vectors, and the notation overload5.1.3 Regression: deblurring as least squares5.1.4 Matrices without forming matrices: gradient descent and conjugate gradient5.1.5 Efficient solvers5.2 Efficient solvers5.2.1 Matrix-free, iterative, multiscale: the situation5.2.2 Gradient descent and conjugate gradient (recap + preconditioning)5.2.3 Multigrid: coarse-to-fine on a pyramid5.2.4 The FFT solver: diagonalize when you can5.2.5 Which solver wins — and why this is the foundation of the part5.3 Information theory5.3.1 The basics, at a high level5.3.2 Compression: the clearest application5.3.3 The imaging system as a channel5.3.4 Information-theoretic quantities as objectives5.3.5 The modern revival: learned compression and generative models5.3.6 Cousins, not children5.3.7 Where it is least relevant: aesthetics and perception5.4 Fundamental limits of inverse problems5.4.1 The question: what can no algorithm recover?5.4.2 The SVD picture: null space, near-null, and the noise explosion5.4.3 Information bounds: a floor no estimator beats5.4.4 The resolution–noise tradeoff, and how a prior invents5.4.5 Fundamental versus practical, and coded capture as the escape5.5 Machine learning5.5.1 The framing: learned operators replace hand-designed ones5.5.2 The data story: synthetic data, noise models, datasets5.6 Deep learning5.6.1 Low-level learned operators (pixel-to-pixel)5.6.2 Mid- and high-level learned predictors5.6.3 Generative models: image-to-image translation5.6.4 Learned perceptual metrics and losses5.7 Generative AI and diffusion5.7.1 The framing: generation is learning and sampling a prior $p(x)$5.7.2 Diffusion: generation as iterated denoising5.7.3 Conditioning: text, images, and control5.7.4 Posterior sampling: generative priors for inverse problems5.7.5 Other generative families, in brief5.7.6 Caveats and ethics5.8 Image priors5.8.1 A catalog of priors: from smoothness to learned5.8.2 Shift and scale invariance5.8.3 Anisotropy: sparsity in a single direction5.8.4 Linear color models5.8.5 Denoising as a universal prior: Plug-and-Play and RED5.8.6 Diffusion is iterated denoising (the continuous limit)5.8.7 Score-based models, the thin manifold, and imperfect priors6 EDGES MATTER
6.1 Bilateral filtering6.1.1 Motivation: local tone-mapping haloes6.1.2 Bilateral filter6.1.3 Cross / joint bilateral6.1.4 Bilateral grid6.1.5 Bilateral-grid learning (HDRnet)6.2 Non-local means6.3 Edge-preserving optimization — colorization6.3.1 The shift: filtering becomes optimization6.3.2 Colorization: the canonical demonstration6.3.3 Writing down the energy6.3.4 Solving it: a sparse linear system we already own6.3.5 One affinity, two faces: the matting Laplacian connection6.4 Local Laplacian filters6.4.1 The wish, and why the bilateral leaves a halo6.4.2 What a Laplacian pyramid hands you6.4.3 The mechanism: a remapping recomputed per output pixel6.4.4 One curve, three edits: enhancement, tone mapping, inverse tone mapping6.4.5 Contrast with the bilateral base/detail split6.4.6 The cost, and the fast version6.5 Matting Laplacian6.5.1 The matting problem: compositing, and why it is under-determined6.5.2 The color-line assumption: α becomes linear in color6.5.3 Closed-form matting: the matting Laplacian6.5.4 Spectral matting: eigenvectors of the matting Laplacian6.5.5 An alternative matting Laplacian6.5.6 Beyond matting: spatially-varying white balance6.6 Guided image filtering6.6.1 The key idea: the output is a local line in the guide6.6.2 Fitting the line, and the one knob6.6.3 Why it is fast: O(N) regardless of window size6.6.4 No gradient reversal6.6.5 Reading the affinity back out6.6.6 Where it is used6.7 Locally adaptive regression kernel (LARK)6.7.1 Change the verb: from averaging to fitting6.7.2 Steering the kernel to the local structure6.7.3 The regression flavor of the affinity6.7.4 LARK as a structure descriptor6.8 Poisson image editing6.8.1 Edit gradients, not pixels6.8.2 Seamless cloning: the headline application6.8.3 Reconstruction = solving the Poisson equation6.8.4 Poisson vs. pyramid blending6.8.5 Advanced techniques6.9 Seam optimization6.9.1 Find the non-edges: least-noticeable cuts6.9.2 Intelligent scissors / live-wire: the cut as a least-cost path6.9.3 Snakes (active contours): the continuous cousin6.9.4 Graph cut: globally optimal boundaries by min-cut / max-flow6.9.5 GrabCut: interactive cutout from a single rectangle6.9.6 Normalized cuts: the spectral relaxation6.9.7 Seam carving: content-aware resize by DP6.9.8 Time-lapse via DP: a seam through time6.9.9 Video textures: looping a seam in time6.9.10 GraphCut textures & photomontage: cut, then blend6.9.11 Where this connects: MRFs and beyond6.10 Recap: which edge-aware technique when?6.10.1 The three relationships to edges6.10.2 Pros, cons, and cost: when to reach for each6.10.3 A worked chooser7 WARPING, MORPHING, AND LAGRANGIAN APPROACHES
7.1 Warping7.1.1 Warping is a domain transform: move *where*, not *what*7.1.2 Forward vs inverse warping: why output-driven wins7.1.3 Resampling: a quick reminder (developed in BASIC)7.1.4 Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladder7.1.5 Specifying the warp II: free-form warps from sparse correspondences7.1.6 Liquify: the warp with a paintbrush bolted on7.2 Antialiasing for complex transforms7.2.1 When the footprint stops being a square: the warp Jacobian7.2.2 EWA: the elliptical weighted average7.2.3 Feline and anisotropic MIP probing: EWA on a GPU budget7.2.4 Other footprint integrators: summed-area tables and ripmaps7.2.5 Footprint estimation, clamping, and when to reach for it7.3 Morphing7.3.1 Why a cross-dissolve isn't enough: the ghosting motivation7.3.2 Two interpolations: domain (shape) and range (color)7.3.3 The morphing recipe (combine both)7.3.4 Field morphing: the Beier–Neely line-pair warp7.3.5 Mesh / triangulation morphing (the alternative warp)7.3.6 View morphing: the geometrically-correct in-between of two views7.3.7 Recap and significance7.4 Morphable models7.4.1 Step 1: dense correspondence, the precondition for averaging shapes7.4.2 Step 2: Procrustes alignment, factor out pose so PCA sees shape7.4.3 Step 3: PCA, a mean and a basis of eigen-deformations7.4.4 Step 4: shape and appearance are *separate* bases7.4.5 Step 5: fitting a new photo, analysis-by-synthesis7.4.6 Step 6: once fitted, edit by moving in the space7.4.7 Step 7: from linear PCA to neural priors7.4.8 Recap and significance7.5 Shape-preserving warping7.5.1 Why plain interpolation shears: the rigidity gap7.5.2 As-rigid-as-possible (ARAP) manipulation7.5.3 Linear blend skinning and why the weights decide everything7.5.4 Bounded biharmonic weights (BBW)7.5.5 ARAP interpolation: rigid-as-possible morphing between two poses7.5.6 Moving least squares: the simpler cousin7.5.7 Recap and significance7.6 Perspective distortion and its correction7.6.1 Keystoning is projection, not a lens flaw7.6.2 The fix is a homography: re-render the façade fronto-parallel7.6.3 The optical alternative at capture: tilt-shift / Scheimpflug7.6.4 Cropping wide-angle photographs: recenter with a homography, don't just crop7.6.5 A different perspective distortion: wide-angle portraits, and a content-aware fix7.6.6 Beyond faces: correcting arbitrary objects (MaDCoW)7.6.7 A montage has no single viewpoint: a family in a box7.6.8 Recomposing perspective after capture: computational zoom7.6.9 The catch: resampling cost, and "only a plane rectifies exactly"7.6.10 Where this sits: one map, then transport8 MATCHING PIXELS AND HUMANS ACROSS SPACE AND TIME
8.1 Brute force8.1.1 Why you must align first8.1.2 Brute-force translational alignment (SSD / NCC)8.1.3 Phase correlation: the whole shift from one FFT8.1.4 Coarse-to-fine alignment on a pyramid8.2 Sub-pixel matching8.2.1 Refining the cost: fit a parabola8.2.2 Refining in the frequency domain: phase correlation8.2.3 Refining by gradient: the Lucas–Kanade step8.2.4 What breaks it: peak-locking, texture, and noise8.2.5 One tool, everywhere downstream8.3 Sparse matching8.3.1 Where to look: corners, and the structure tensor8.3.2 Invariance: surviving scale and rotation8.3.3 Describing a neighborhood: SIFT and its zoo8.3.4 Matching: nearest neighbor in descriptor space8.4 Feature tracking8.4.1 Tracking vs dense flow: sparse-but-long vs dense-but-short8.4.2 KLT = Lucas–Kanade, per feature, iterated over time8.4.3 Good features to track = where the structure tensor is well-conditioned8.4.4 What breaks long-term tracking: drift, appearance change, occlusion, and re-detection8.4.5 Modern point trackers (briefly)8.4.6 An application: synthetic motion blur from a track8.5 Robustness: the ratio test and RANSAC8.5.1 The ratio test: reject ambiguous matches before fitting8.5.2 RANSAC: fit from a minimal random sample, score by consensus8.5.3 Variants and degeneracy8.6 Deep learning approaches to sparse matching8.6.1 Learned local features: SIFT's job, done by a network8.6.2 Learned matching — reason about the whole set at once8.6.3 The 3-D-aware turn: pointmaps subsume matching8.7 Fast matching8.7.1 Approximate nearest neighbors for sparse descriptors8.7.2 PatchMatch: randomized dense correspondence by propagation8.7.3 Fast high-dimensional matching by random projection8.8 Optical flow8.8.1 What optical flow is: and whether it is even well-defined8.8.2 Brightness constancy and the optical-flow constraint8.8.3 The aperture problem8.8.4 Lucas–Kanade: local constant-flow least squares (and the structure tensor)8.8.5 Horn–Schunck: global smoothness regularization8.8.6 Large motion: coarse-to-fine warping8.8.7 Learned flow: RAFT (neuralize the classical pipeline)8.9 Deep learning approaches to optical flow8.9.1 The unrolling principle: neuralize the classical solver8.9.2 Cost volumes and warping inside the net8.9.3 RAFT and the recurrent update8.10 Face tracking8.10.1 Detecting the face8.10.2 Landmarks: pinning down the features8.10.3 Tracking across time8.10.4 Lifting to 3-D: the morphable model8.10.5 Recognition, and the dark side8.10.6 Which library to use8.11 Body pose estimation8.11.1 Top-down vs bottom-up8.11.2 On-device, real time8.11.3 Lifting to 3-D: parametric bodies8.11.4 Which library to use9 SINGLE IMAGE COMPUTATIONAL PHOTOGRAPHY
9.1 Denoising9.1.1 A reminder: bilateral filtering and wavelet/pyramid shrinkage9.1.2 BM3D: group similar patches and filter them together9.1.3 Learning to denoise9.2 Super-resolution9.2.1 What problem super-resolution solves (and why it's ill-posed)9.2.2 Scenarios: single-image, burst, and hybrid space–time9.2.3 Reconstruction vs hallucination: measured detail vs invented detail9.3 Demosaicking and joint reconstruction9.3.1 The pipeline problem9.3.2 FlexISP: one energy, one prior9.3.3 Learning the joint prior: the founding formulation9.3.4 Model-based and unrolled networks9.3.5 Learning where the hard cases are9.3.6 Co-designing the mosaic: a learnable sensor9.3.7 The end-to-end learned ISP9.3.8 Better backbones: CNN → transformer → diffusion prior9.3.9 Exotic mosaics: learned remosaicking9.3.10 Where do the labels come from?9.3.11 Correcting chromatic aberration jointly9.3.12 Recap9.4 Non-blind deblurring9.4.1 Deblurring in the presence of noise: why naive inversion fails9.4.2 The Wiener filter — the regularized, noise-aware inverse9.4.3 Sparse gradients: the prior that keeps edges sharp9.5 Blind deblurring9.5.1 Blind deblurring: estimating the kernel *and* the image9.5.2 A more realistic blur model: spatially-varying (camera-shake) blur9.5.3 Engineering the aperture: depth and all-focus from a coded mask9.6 Dehazing9.6.1 Dehazing as a prior-driven inverse problem9.6.2 Differentiable image pipelines and algorithm optimization (Halide)9.7 Mixed-lighting white balance9.7.1 Why a global gain must fail9.7.2 Estimating the per-pixel mixture: an under-determined inverse problem9.7.3 Correcting each region for its own light9.7.4 What came after9.8 Inpainting, texture synthesis, and object removal9.8.1 Inpainting as filling unmeasured pixels — the spectrum of priors9.8.2 PDE / diffusion-based inpainting9.8.3 Texture synthesis (Efros–Leung; Efros–Freeman quilting)9.8.4 Exemplar inpainting: clone, healing brush, and object removal (Criminisi)9.8.5 Data-driven scene completion (Hays & Efros)9.8.6 Deep inpainting (context encoders → partial/gated conv → diffusion)9.8.7 Highlight / specular recovery9.8.8 Epitomes — a compact patch model9.9 Patch match9.9.1 The nearest-neighbor field, and why exhaustive search is the bottleneck9.9.2 PatchMatch — randomized correspondence [@barnes-etal-2009|Barnes et al. 2009]9.9.3 Applications: hole filling, retargeting, reshuffle9.9.4 Shift-Map image editing [@pritch-etal-2009|Pritch et al. 2009] — editing as graph-cut labeling9.10 Colorization9.10.1 One channel in, three channels out9.10.2 A spectrum of priors: scribbles, references, and learned models9.10.3 The multimodality trap: why naïve colorization goes muddy9.10.4 Closing the loop: learned priors with a human's hints9.10.5 Plausible is not correct9.10.6 Where it sits9.11 Compositing, segmentation and matting9.11.1 Compositing and alpha channels9.11.2 Segmentation: cutting the object out9.11.3 The fundamental matting equation9.11.4 Blue/green-screen matting and chroma keying9.11.5 Traditional matting approaches9.11.6 Deep-learning matting9.11.7 Generative matting9.11.8 Harmonization and where the blends live9.11.9 Optical effects beyond alpha9.12 Illumination related effects in a single image9.12.1 Intrinsic images: the unifying frame ($I = R\cdot S$)9.12.2 Multiple-light / spatially-varying white balance9.12.3 Reflection removal — pulling apart a transmission and a glass reflection9.12.4 Shadow detection and shadow removal9.12.5 Specular-highlight removal / "fake polarization" (the dichromatic model)9.13 Tone Mapping9.13.1 Global vs local, re-hashed9.13.2 A smarter global curve: histogram adjustment9.13.3 A taxonomy of local methods9.13.4 The darkroom ancestor: dodge & burn and the Zone System9.14 Style transfer9.14.1 Classical style transfer: patches and statistics9.14.2 Neural style and feed-forward stylization9.14.3 Style transfer as image-to-image translation9.15 Non-photorealistic rendering9.15.1 What NPR is for, and the one idea9.15.2 Stroke-based / painterly rendering (and the brush p-set)9.15.3 Edge-preserving abstraction: bilateral + Difference-of-Gaussians9.15.4 Example-based stylization and the bridge to neural style9.15.5 Region-based stylization: stained glass, low-poly, mosaics9.15.6 Artistic screening and halftoning10 COMPOUND LENSES, AND ABERRATION CORRECTION
10.1 Aberrations and optical challenges10.1.1 Taxonomy of challenges10.1.2 Spherical aberration10.1.3 Coma10.1.4 Astigmatism10.1.5 Field curvature10.1.6 Chromatic aberration10.1.7 Radial distortion10.1.8 Wave effects and diffraction10.1.9 Vignetting10.1.10 Flare and coating10.2 Aberrations correction10.2.1 The two families of cure10.2.2 Correction in glass10.2.3 Computational correction10.2.4 Radial distortion correction10.3 Measuring lens quality10.3.1 The Modulation Transfer Function (MTF)10.3.2 Measuring MTF in practice10.3.3 Spot diagrams and the PSF10.3.4 MTF through focus and depth of focus10.3.5 Field-dependent MTF: sagittal vs tangential10.3.6 Color, geometry, and illumination measurements10.3.7 Scalar quality summaries10.4 Lens optimization10.4.1 The high-level idea: design as optimization10.4.2 The forward model: ray-tracing and spot diagrams10.4.3 From hand calculation to software10.4.4 Tradeoffs: the design is always a compromise10.4.5 Tolerancing: from the nominal design to a manufacturable one10.5 A short bestiary of classic designs10.5.1 The lens as a system: cardinal points, pupils, f-number, T-stop10.6 Scaling laws in optics10.6.1 Lohmann's scaling laws: why a good lens is heavy10.6.2 The gigapixel barrier for a single lens10.6.3 The escape: monocentric multiscale optics10.6.4 Capture everything, crop later: the spatial cousin of the light field10.7 Special optics10.7.1 Tilt-shift and the Scheimpflug principle10.7.2 Fisheye and non-rectilinear projection10.7.3 Mirrors: catadioptric and reflecting systems10.7.4 Periscope / folded-lens design (smartphone telephoto)10.7.5 Anamorphic optics10.7.6 Stereo (3D) lenses10.7.7 Macro, microscope objectives, and telescopes10.7.8 Teleconverters10.7.9 From shaped glass to thin structures: Fresnel, diffractive, GRIN, metalenses10.8 Focus10.8.1 Focus mechanics10.8.2 Focusing a compound lens: beyond unit focusing10.8.3 Focus actuators10.8.4 Focus stacking, macro, and focusing rails10.9 Autofocus10.9.1 Contrast-detection AF10.9.2 Phase-detection AF (the split-pupil / stereo trick)10.9.3 On-sensor PDAF and dual-pixel AF10.9.4 Depth from focus / defocus, and learned subject AF10.9.5 Where to focus: saliency, faces, and eyes10.9.6 Focusing in astrophotography10.10 Bokeh, focus stacking, and depth-of-field control10.10.1 Recap: the geometry of focus (pointer, not re-derivation)10.10.2 The bokeh look: shape and structure of the blur10.10.3 Extending DoF: focus stacking10.10.4 Controlling and faking DoF10.11 Fake (synthetic) depth of field10.11.1 Why phones must fake it10.11.2 Where the depth (or subject) comes from10.11.3 From depth to blur — the thin-lens circle10.11.4 Realistic bokeh — why a Gaussian looks fake10.11.5 Occlusion-aware compositing and matting10.11.6 Failure modes10.12 Glare suppression10.12.1 Where stray light comes from: flare, ghosting, veiling glare10.12.2 Hardware suppression10.12.3 Computational deflare and glare deconvolution10.13 Optical stabilization10.13.1 The problem: hand-shake and the blur budget10.13.2 Optical stabilization: lens-shift vs. sensor-shift (IBIS)10.13.3 Digital / electronic stabilization and the computational alternatives10.14 The eye as an optical instrument: vision and its correction10.14.1 The eye as a camera10.14.2 Refractive errors: the eye out of focus10.14.3 Correcting vision10.14.4 Presbyopia and the bifocal problem: from bifocals to AF glasses11 MULTIPLE EXPOSURE IMAGING
11.1 Denoising by averaging11.1.1 Why averaging works — the $1/\sqrt N$ derivation11.1.2 When the plain mean is wrong: robust combination11.1.3 Calibration frames: what averaging can't fix11.1.4 Handheld low light: the phone in your pocket11.1.5 Deep-sky astrophotography: averaging at the extreme11.2 HDR merging11.2.1 The HDR challenge11.2.2 Data capture: how to vary the exposure11.2.3 Curve calibration11.2.4 Combining exposures11.2.5 Optimizing the capture and merge11.2.6 In-sensor HDR: dynamic range without a bracket11.3 Application to cell phones: HDR+ and burst imaging11.3.1 Why a phone shoots a burst, and why it underexposes11.3.2 The HDR+ pipeline: align and robust-merge in raw11.3.3 From burst HDR to burst super-resolution11.4 Multiframe or burst super-resolution11.4.1 The reconstruction principle: many coarse grids make one fine grid11.4.2 Sub-pixel registration is the whole game11.4.3 Fusion: from scattered samples to a sharp image11.4.4 Accidental versus deliberate offsets: hand tremor and pixel-shift sensors11.4.5 The learned era11.4.6 Where it runs out11.5 Manual panorama stitching from multiple views11.5.1 The scenario, and the one rule: rotate, don't translate11.5.2 Refresher: pinhole projection is "divide by depth"11.5.3 Why you don't need 3D: depth cancels for a pure rotation11.5.4 Homographies and homogeneous coordinates11.5.5 Solving for $H$ from correspondences11.5.6 Warping and assembling the panorama11.5.7 Another application: document flattening and merging11.6 Automatic panorama stitching from multiple views and feature matching11.6.1 Why not brute force, and the two sub-problems11.6.2 The feature pipeline, recalled from Part 711.6.3 RANSAC for a homography11.7 Blending11.7.1 Why a hard seam is visible — the photometric mismatch11.7.2 Feathering / alpha blending — and why it ghosts11.7.3 Two-scale blending — the simple split (the pset method)11.7.4 Multiband / Laplacian-pyramid blending — a transition per band11.7.5 Poisson / gradient-domain blending — paste gradients, solve for values11.7.6 Seam optimization — route the seam instead of fading it11.7.7 The complete pipeline, end to end11.8 Bells and whistles11.8.1 Other projections11.8.2 Bundle adjustment11.8.3 Movement and parallax handling11.9 Continuous panoramas (e.g. on cell phones)11.9.1 Incremental registration of a video stream11.9.2 Mosaicking a moving strip (and why a *central* strip)11.9.3 Rolling shutter and exposure drift11.10 Focal stacks and depth of field extension11.10.1 Why limited depth of field is the problem11.10.2 A simple algorithm: sharpness = local high-frequency energy, then argmax11.10.3 The more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)11.10.4 Capturing the stack — hardware, and the magnification trap11.11 Hyperspectral imaging, color wheels11.11.1 Why three numbers aren't enough — RGB as a 3-sample projection11.11.2 Building the spectral stack — filter wheels, tunable filters, pushbroom, snapshot11.11.3 What it's for — material ID, agriculture, art and beyond11.12 Polarization imaging11.12.1 What a camera throws away — polarization as a third axis of light11.12.2 Capturing the stack — rotate a polarizer, or a polarization mosaic11.12.3 Combining images at different polarizations11.13 Intrinsic images with time lapse11.13.1 The split, and why one image can't do it11.13.2 Weiss 2001 — the median of log-gradients11.13.3 Where this chapter belongs — passive vs. active illumination11.14 Lucky imaging (planetary / lunar astro)11.14.1 Atmospheric seeing — why one long exposure fails11.14.2 Shoot thousands, keep the sharpest11.14.3 Align and stack the survivors11.14.4 A poor man's adaptive optics12 MANY IMAGES AND PHOTO COLLECTIONS
12.1 Photo Mosaics12.1.1 The tiling-and-matching pipeline12.1.2 Color correction and avoiding repeats12.1.3 Multi-scale and irregular tilings12.1.4 Why it resolves into the target at a distance12.2 Retrieval12.2.1 Classic CBIR — histograms and the text-retrieval analogy12.2.2 Deep retrieval — learned embeddings and CLIP12.2.3 Retrieval at scale — approximate nearest neighbor12.2.4 Mining what makes a place distinctive12.3 Auto curation12.3.1 Technical quality — the easy rejects12.3.2 Aesthetics — the hard, learned part12.3.3 Grouping, summary, and diversity12.3.4 Arranging the collection, not just culling it12.4 Life logging cameras12.4.1 The devices and how they fire12.4.2 The memory-prosthesis reframing12.4.3 The big-data problem passive capture creates12.4.4 Privacy and ethics12.5 Inpainting Using Millions of Photographs12.5.1 Why self-similar inpainting isn't enough12.5.2 Scene completion from a huge database12.5.3 The data is the prior — and its modern opposite12.6 Photo tourism12.6.1 Structure-from-motion on internet collections12.6.2 From reconstruction to experience12.6.3 Lineage12.7 Photobios12.7.1 Align and order the collection12.7.2 Let the data fill the gaps12.7.3 Collection as experience12.8 Average Explorer12.8.1 Galton's composite portraiture — and what it was for12.8.2 Alignment is everything12.8.3 The average of a category12.8.4 An artistic lineage12.8.5 AverageExplorer: averaging made interactive12.8.6 What an average is, and what it is good for12.9 Pix 2 GPS12.9.1 Geolocation as retrieval over a geotagged corpus12.9.2 The answer is a distribution12.9.3 Mapping the collection itself12.9.4 The learned successors12.10 Personalized priors12.10.1 Personalized restoration12.10.2 Personalizing generative models12.10.3 The bargain and its ethics12.11 Artistic projects with photo collections12.11.1 Statistical collage — Salavon12.11.2 Anticliché camera12.12 Pareidolia12.13 Displaying images together12.13.1 Selection12.13.2 Layout12.13.3 Color and coherence12.13.4 Themes13 VIDEO
13.1 Motion blur, temporal sampling, and resampling13.1.1 A frame is an integral over time → motion blur13.1.2 Time is sampled → temporal aliasing, the wagon-wheel effect13.1.3 Motion blur *is* the temporal prefilter: the two are one tradeoff13.1.4 Resampling in time: frame-rate conversion13.1.5 Lagrangian vs Eulerian: the organizing distinction for the part13.2 Video compression and motion compensation13.2.1 Why video compresses far better than still × N: temporal redundancy13.2.2 Motion-compensated prediction: the core trick13.2.3 I, P, and B frames; GOP structure13.2.4 Why this is "optical flow on a budget"13.2.5 Modern codecs in one breath13.3 Video editing13.3.1 Non-linear editing: the timeline metaphor13.3.2 Summarization: keyframes, fast-forward, and highlights13.3.3 Fun temporal filters: reduce-over-time13.3.4 Transcript-based editing13.3.5 In-betweening: synthesizing the frames an edit needs13.3.6 Coda: storyboards, interviews, and where this part lands13.4 Frame interpolation and slow-motion synthesis13.4.1 Why interpolate: faking slow-motion and up-converting frame rate13.4.2 Interpolation = morphing between adjacent frames13.4.3 Flow-based interpolation: warp both frames to the midpoint and blend13.4.4 Learned synthesis: Super SloMo and FILM13.5 Hybrid low/high resolution and frame-rate imaging13.5.1 The space–time bandwidth trade-off13.5.2 The hybrid two-camera architecture13.5.3 Motion from the fast stream13.5.4 Applications: deblur, space–time super-resolution, video from stills13.5.5 Modern descendants13.6 Video stabilization and rolling-shutter correction13.6.1 What stabilization is: a camera-path signal to be smoothed13.6.2 Stage 1: estimating the camera trajectory13.6.3 Stage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)13.6.4 Stage 3: re-rendering and the stabilization↔crop tradeoff13.6.5 Rolling-shutter correction: per-row pose and rectification13.7 Time-lapse photography13.7.1 Capturing a time-lapse: interval, shutter, and day-to-night transitions13.7.2 Deflickering: stabilizing exposure, white balance, and light13.7.3 Hyperlapse: stabilizing a time-lapse that also moves13.7.4 Mining time-lapses from the internet13.7.5 Factoring a time-lapse: reflectance, illumination, and intrinsic images13.8 Video textures13.8.1 Finding good transitions13.8.2 Playing it13.8.3 Relatives and descendants14 LIGHT FIELDS AND PLENOPTIC CAMERAS
14.1 Light fields 10114.1.1 Capture rays, not pixels14.1.2 The plenoptic function, reduced to four dimensions14.1.3 Ray and point are dual14.1.4 Reading the 4-D structure through its 2-D slices14.1.5 Rendering a new view by looking up rays14.1.6 Light fields vs. plenoptic and radiance14.2 Light field cameras14.2.1 Integral photography: Lippmann's fly's-eye plate (1908)14.2.2 The plenoptic camera: a microlens array on the sensor14.2.3 Lytro: the consumer plenoptic camera14.2.4 Commercial light-field cameras beyond Lytro14.2.5 Camera arrays: a grid of full cameras14.2.6 The spatial↔angular tradeoff14.2.7 Two strategies, and why arrays and microlenses are dual14.3 Other light field acquisition setups14.3.1 One camera on a gantry: sample the aperture in time14.3.2 Handheld, unstructured capture: let the poses be irregular14.3.3 Catadioptric capture: one sensor, many viewpoints at once14.3.4 Coded aperture in time: sweep the pupil itself14.3.5 The everything-else, mapped14.4 Refocusing and synthetic aperture14.4.1 Reconstructing a photo: it is all about which rays you sum14.4.2 Digital refocusing is shift-and-add14.4.3 A focal stack from one capture, and an all-in-focus image14.4.4 Fourier-slice photography: the fast version14.4.5 Reading refocus and depth off an epipolar slice14.4.6 Synthetic aperture: an aperture the size of a room14.5 Aberration correction in light fields14.5.1 An aberration is misrouted rays14.5.2 Re-routing each ray to the ideal-lens position14.5.3 The trade: ray bookkeeping instead of glass, and only what you sampled14.6 Light field aliasing and 4D Fourier analysis14.6.1 The light field is a sampled signal, and its samples are viewpoints14.6.2 Depth is slope is spectral orientation14.6.3 The bowtie: a spectrum shaped by the scene's depth range14.6.4 How densely must you sample? The plenoptic-sampling bound14.6.5 Geometry buys back samples: the depth-vs-views tradeoff14.6.6 Under-sampling looks like a ghost14.6.7 Where this sits14.7 Lumigraph and shape priors for sharper light field rendering14.7.1 Pure light-field rendering blurs because it has no shape14.7.2 The Lumigraph: reproject onto a geometry proxy, then blend14.7.3 Unstructured inputs: free-hand views, no grid required14.7.4 Surface light fields14.8 Light field microscopy14.8.1 The optical setup: a microlens array at the intermediate image plane14.8.2 The spatial-versus-angular trade: a coarse 3-D volume14.8.3 Recovering the volume: synthetic refocusing, then 3-D deconvolution14.8.4 Why single-shot 3-D is the whole point14.9 Light field networks14.9.1 NeRF: a radiance field rendered by volume integration14.9.2 Light field networks: a ray straight to color, in one evaluation14.9.3 A family of neural light fields14.9.4 Test-time training: the network as the per-scene prior14.9.5 Hand-off: from rays to radiance fields and generation14.10 Practical aspects of light field cameras14.10.1 Do you lose all that resolution?14.10.2 Can you do video? Is it practical? The bandwidth problem14.10.3 The time dimension is high speed14.10.4 Adjacent frontiers, briefly14.10.5 So, do I get my camera?15 MULTI-APERTURE IMAGING
15.1 Camera arrays: one rig, many instruments15.1.1 One rig, four instruments15.1.2 Commercial arrays: the Light L1615.1.3 Where this sits15.2 Bullet time15.3 Multi-camera phones16 COMPUTATIONAL OPTICS AND CODED IMAGING
16.1 Wavefront coding16.1.1 Why you cannot just deblur defocus16.1.2 The fix: re-engineer the blur16.1.3 The cubic phase plate16.1.4 The depth-invariant PSF and a single deconvolution16.1.5 What it costs16.1.6 Where it sits16.2 Compressive sensing16.2.1 Sub-Nyquist: fewer measurements than unknowns16.2.2 Two ingredients: incoherent measurements and sparsity16.2.3 Recovery by $\ell_1$: the geometry of basis pursuit16.2.4 Why it works: the restricted isometry property, intuitively16.2.5 The single-pixel camera16.2.6 Where compressive sensing pays off — and where it does not16.3 Coded aperture16.3.1 The bad forward operator of a clear aperture16.3.2 A mask designed for a flat, zero-free spectrum16.3.3 One shot, two outputs: depth and an all-in-focus image16.3.4 The design criterion: which pattern?16.3.5 Coded-aperture pairs: splitting the trade across two shots16.3.6 Heterodyning the light field: dappled photography16.3.7 Where it sits16.4 Phase-coded apertures16.4.1 A reminder on wavefront coding16.4.2 Focus sweep16.4.3 The lattice-focal lens16.4.4 Where it sits16.5 Code in time (phase, amplitude)16.5.1 Why ordinary motion blur is (almost) unrecoverable16.5.2 The flutter shutter: chop the exposure into a code16.5.3 The decode: one deconvolution, a sharp moving object16.5.4 Amplitude in time, and phase in time16.5.5 Coded strobing, temporal multiplexing, and compressive video16.5.6 Motion-invariant photography16.5.7 Where it sits16.6 Theoretical analysis of imaging systems in the 4D light field Fourier domain16.6.1 The light field's spectrum, and the one move that explains everything16.6.2 Every camera is a different slice16.6.3 Putting cameras on one footing: the Bayesian comparison16.6.4 The lattice-focal lens: tiling the wedge16.6.5 The upper bound — and the gap we have not closed16.6.6 From cameras to light transport: the same spectral lens16.6.7 Where this leaves the part16.7 End-to-end optimization16.7.1 The pipeline as one differentiable graph16.7.2 Backpropagating into the glass16.7.3 What gets designed: a height map, not a hyperparameter16.7.4 A gallery of deep-optics results16.7.5 What it costs, and where it can go wrong16.7.6 Where it sits16.8 Fourier optics16.8.1 Light as a wave: amplitude, phase, and what "coherent" means16.8.2 Diffraction is a Fourier transform16.8.3 A lens computes a Fourier transform16.8.4 The pupil function *is* the transfer function16.8.5 The diffraction limit16.8.6 Aberrations are pupil phase16.8.7 Every code in this part is a choice of pupil16.8.8 Fourier ptychography: synthesizing a bigger pupil16.9 Exotic / advanced opticsoutline 16.9.1 Lensless imaging16.9.2 GRIN16.9.3 Metalenses and advanced crazy optics à la Barbastathis (coherent though)16.9.4 Non-linear optics16.10 Optical modulators (spatial light modulators): DMD and LCD/LCoSoutline 17 COMPUTATIONAL SENSORS
17.1 Assorted pixels17.1.1 Dual-pixel and phase-detect pixels: buying depth and focus17.1.2 Clear, white, and other color-filter variants17.1.3 Polarization pixels17.1.4 Spatially varying exposure: assorting for dynamic range17.1.5 The common thread17.2 Modern sensors (quad Bayer, in-sensor HDR, and beyond)17.2.1 Quad Bayer, Tetracell, and nona-binning17.2.2 In-sensor HDR: capturing range before the merge17.2.3 Dual-pixel autofocus, on the same sensor17.2.4 BSI and stacked sensors: compute under the pixels17.2.5 The global-versus-rolling shutter trade17.2.6 Beyond: nano-prism, organic, and event pixels17.3 On-sensor HDR17.3.1 Staggered / multiple-exposure readout (DOL-HDR)17.3.2 Dual (and triple) conversion gain (DCG)17.3.3 Split-pixel: a large and a small photodiode17.3.4 Spatially-varying exposure (SVE) / assorted exposures17.3.5 Lateral overflow integration capacitor (LOFIC)17.3.6 Logarithmic, self-resetting, and counting pixels17.3.7 On-sensor HDR versus multi-frame HDR17.4 Depth sensors17.4.1 Stereo: passive triangulation17.4.2 Structured light: projecting the texture17.4.3 LiDAR and direct time-of-flight17.4.4 Time of flight17.4.5 Passive depth from one camera, in passing17.4.6 The menu, in one view17.5 Single-photon sensors (SPAD, avalanche, photon counting)17.5.1 From avalanche gain to a single-photon click17.5.2 Photon-counting arrays: zero read noise, shot-noise-limited17.5.3 The Quanta Image Sensor: a different road to one photon17.5.4 What it costs: dark counts, dead time, fill factor, data rate17.6 Doppler / velocity imaging17.6.1 The Doppler shift, as a velocity sensor17.6.2 The instruments: vibrometry, Doppler LiDAR, radar17.6.3 Where it fits17.7 Event sensors17.7.1 How it works: per-pixel change detection17.7.2 The upside: microseconds, dynamic range, no blur, little data17.7.3 The downside: no picture, and a new kind of data17.7.4 Uses, lineage, and the contrast with single-photon17.8 Specialized and research sensors17.8.1 Geiger-mode avalanche-photodiode arrays: photon-counting laser radar17.8.2 Digital-pixel focal-plane arrays17.8.3 Scientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaics17.8.4 Stacked and processing-in-pixel sensors17.8.5 Curved focal planes17.8.6 Beyond the visible, and filter-array sensors17.8.7 Radiation-hardened and defense focal planes17.8.8 The chapter's point17.9 Extra sensors and non-visual data17.9.1 Accelerometer and gyroscope: the inertial measurement unit17.9.2 Sound: microphones, audio-visual sync, and the visual microphone17.9.3 GPS: geotagging and place17.9.4 Compass and magnetometer: heading and orientation17.9.5 Near-infrared: the cut filter, dark flash, and NIR-assisted denoise17.9.6 Temperature: dark-current compensation17.10 Ultra High speed Imaging17.10.1 Streak cameras: sweeping time onto a spatial axis17.10.2 Femto-photography: a movie of light in flight17.10.3 Compressive ultrafast photography: a single-shot coded streak17.10.4 Transient imaging and looking around corners17.10.5 The bridge to direct time-of-flight and LiDAR18 COMPUTATIONAL ILLUMINATION
18.1 Flash photography18.1.1 Flash / no flash18.1.2 Ramesh's multiflash18.1.3 Removing flash artifacts18.1.4 Dark flash (plus Stasi version!)18.2 High-speed and stroboscopic photography18.2.1 Freezing motion: the microsecond strobe18.2.2 The trigger problem18.2.3 Stroboscopic multiplicity: a sequence on one frame18.2.4 Digital descendants: LED strobes and high-speed cameras18.3 Illumination-based matting18.3.1 The well-posed case to beat: chroma key18.3.2 The magic prism: Disney's sodium-vapor process18.3.3 Near-infrared and time-multiplexed matting18.3.4 Flash/no-flash matting: separation by falloff18.3.5 The throughline: control the capture, not the prior18.4 Separation of Direct and Global Illumination18.4.1 The frequency insight18.4.2 Nayar's program: programmable, structured illumination18.4.3 Toward coherent separation18.5 Light domes18.5.1 The reflectance field and one-light-at-a-time capture18.5.2 Relighting by linear combination18.5.3 Scaling down: tabletop LED domes18.5.4 Scaling out: the dome taken into the wild18.5.5 The found dome: the eye18.6 Automatic aesthetic lighting18.6.1 Computational bounce flash18.6.2 Drone lighting: flying the light into place18.7 Dual photography18.7.1 Light transport as a matrix18.7.2 Helmholtz reciprocity: transpose the matrix18.7.3 The unsettling reach: privacy and seeing the unseen18.8 Coherent imaging18.8.1 The confocal principle: rejecting out-of-focus light18.8.2 Seeing through scattering media18.8.3 The part in one line19 3D AND DEPTH
19.1 Multiple view geometry19.1.1 Two views: stereo and disparity19.1.2 Two-view geometry: epipolar lines, and the essential and fundamental matrices19.2 What "depth" means, and where it comes from19.2.1 Depth is the *z*-coordinate, not the ray length19.2.2 Where depth comes from: a cue-and-sensor inventory19.2.3 Relative vs metric: the scale you usually don't have19.3 Monocular depth estimation (one image → depth)19.3.1 Why one image cannot determine depth19.3.2 Relative depth, and the scale-and-shift ambiguity19.3.3 From hand-built priors to borrowed diffusion priors19.3.4 What the maps are good for19.4 Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns19.4.1 The universal recipe, and why holes are the hard part19.4.2 Tour Into the Picture: the spidery mesh19.4.3 Automatic Photo Pop-up19.4.4 3-D Ken Burns and 3-D photos: the monocular form19.5 Multi-view 3-D reconstruction: the classic pipeline19.5.1 The pipeline, stage by stage19.5.2 Why it is brittle: and why SfM survives anyway19.6 Structured light scanning19.6.1 Projector as inverse camera: triangulation with trivial correspondence19.6.2 The coding ladder, and the frames-versus-motion trade19.6.3 Calibration and failure modes19.7 Photos → radiance fields and Gaussian splatting (NeRF, 3DGS)19.7.1 Two goals, one diagram19.7.2 Inverse differentiable rendering, and the discontinuity that forced fuzziness19.7.3 NeRF: a scene as a tiny neural network19.7.4 Do we even need the network?19.7.5 3-D Gaussian Splatting: the lessons without the deep learning19.7.6 The practical recipe: and what is baked in19.7.7 Relaxing the assumptions: NeRF in the wild19.8 Feed-forward (amortized) 3-D: skip the per-scene optimization19.8.1 Amortization: pay once, reuse forever19.8.2 The line: DUSt3R, MASt3R, VGGT19.8.3 The punchline, and the loop it closes19.8.4 The trade, and the data dependency19.9 Re-photography19.9.1 Why you cannot just overlay19.9.2 The real-time guidance loop19.9.3 Where it sits19.10 The landscape, and is 3-D a "fake task"?19.10.1 The field as a landscape, not a line19.10.2 Is 3-D a "fake task"?19.10.3 The counterpoint, and the frontier20 INTEGRAL AND IMMERSIVE IMAGING
20.1 Stereo glasses20.1.1 Wheatstone's stereoscope (1838)20.1.2 Routing a different image to each eye20.1.3 Shooting and synthesizing a stereo pair20.2 VR goggles20.2.1 The three levels20.2.2 The optics: a microdisplay and a magnifier per eye20.2.3 Tracking, latency, and why headsets used to make people sick20.2.4 Passthrough, mixed reality, and where today's products sit20.3 3D displays with accommodation20.3.1 Four ways to deliver a focus cue20.3.2 The other gaps, and the ultimate display20.4 Lenticular displays20.4.1 Lippmann's integral photography: the common ancestor20.4.2 Multi-view, and the resolution–views tradeoff20.4.3 Light-field telepresence: Google Starline20.4.4 Display depth of field and antialiasing20.5 Holography20.5.1 Lippmann and Gabor: recording the wave20.5.2 Off-axis holography and the space-bandwidth wall20.5.3 Computational holography20.6 Retinal projection20.6.1 The Maxwellian view: focus set by the display, not the eye20.6.2 From the virtual retinal display to laser eyewear20.6.3 The frontier: writing to individual cones21 REVEALING THE INVISIBLE
21.1 Accidental cameras21.1.1 The accidental pinhole: a window is a camera21.1.2 The accidental pinspeck: the anti-pinhole21.1.3 The occluder as a crude lens, and recovery as deconvolution21.1.4 Corners and doorways: an edge that resolves the hidden room21.1.5 Where else the world hides a camera21.2 Reflections in the eye21.2.1 The cornea as a catadioptric mirror21.2.2 The geometry: from a corneal pixel to a direction in the world21.2.3 What the recovered reflection is good for21.2.4 Eyes for relighting21.3 Motion and video magnification21.3.1 The Lagrangian precursor: track, then exaggerate21.3.2 Eulerian video magnification: amplify the time series at each pixel21.3.3 Why amplifying brightness amplifies motion21.3.4 Phase-based magnification: move the motion into phase21.3.5 What it reveals: vital signs, structures, materials, modes21.4 Visual microphone21.4.1 From sub-pixel motion to a sound waveform21.4.2 Bandwidth: high-speed cameras and the rolling-shutter trick21.4.3 How good is the copy? The object's frequency response21.4.4 The active cousins: laser vibrometry and interferometry21.5 Corner camera21.5.1 The edge as a one-dimensional aperture21.5.2 From a faint gradient to a usable signal21.5.3 What the corner can and cannot tell you21.6 Active non-line-of-sight21.6.1 Third-bounce geometry and time-of-flight21.6.2 The hardware: photographing light in flight21.6.3 From back-projection to fast, exact inversion21.6.4 The trade, stated plainly21.7 Passive non-line-of-sight21.7.1 The occluder is what makes it solvable21.7.2 A deconvolution where the lens is unknown21.7.3 One dimension: the corner camera21.7.4 Two dimensions from a single photo: computational periscopy21.7.5 Active versus passive, the ledger21.8 Mm-wave, wifi21.8.1 Why radio walks through walls21.8.2 Time of flight, again — radar is NLOS with a longer wave21.8.3 From a radio smear to a human skeleton: the learned map21.8.4 What it sees, and what it costs us22 ADJACENT FIELDS AND APPLICATIONS
22.1 Optical computingoutline 22.2 Astrooutline 22.2.1 Extreme long exposure22.2.2 Tracking, stacking, and selection (pointers)22.3 X-rayoutline 22.4 Medicaloutline 22.5 Microscopyoutline 22.6 Mm-waveoutline 22.7 Music, soundoutline 22.8 Fluorescenceoutline 22.9 Opto-acousticoutline 22.10 Ultrasoundoutline 22.11 Aerial imagingoutline 22.12 Computer visionoutline 22.13 Robotics, drivingoutline 23 HUMAN FACTORS
23.1 Human factors and the art of photography23.1.1 Make better photos23.1.2 Typical shooting scenarios23.1.3 Macro photography23.1.4 Special effect photography23.1.5 Fun artsy stuff23.1.6 Perception of art23.2 Ethics of computational photography23.3 Computational models of perceptionoutline 23.3.1 Spatial (and spatio-temporal) vision23.4 User studiesoutline 23.5 Accessibility: photography by and for blind usersoutline 23.5.1 Blind camera — capture without a sighted operator23.6 The social and personal practice of photographyoutline 24 IMAGE FORENSICS AND AUTHENTICATION
24.1 Image Forensicsoutline 24.1.1 The problem and the threat model24.1.2 Sensor and pipeline traces: PRNU, CFA, and noise24.1.3 Compression, geometry, and metadata forensics24.1.4 Deepfakes, GAN/diffusion fingerprints, and learned detection24.1.5 Why forensics is evidence, not proof — and the hand-off to provenance24.2 Authentication and Provenance (C2PA)24.2.1 From detection to attestation: trustworthy cameras and watermarking24.2.2 C2PA and Content Credentials: the standard24.2.3 AI disclosure, watermarking, and the regulatory push24.2.4 Limits, critiques, and the forensics partnership25 SYSTEMS
25.1 Programmable and modular camerasoutline 25.2 Image processing librariesoutline 25.3 Lightroom-style raw developersoutline 25.4 Photoshop-style editorsoutline 25.5 Networking and image transportoutline 25.6 Photography programming on phonesoutline 26 PERFORMANCE ENGINEERING AND HALIDE
26.1 8-bit and fixed-point arithmetic26.1.1 Integer versus fixed-point: where the binary point sits26.1.2 Rounding, dithering, and the banding trap26.1.3 Saturation and the width of the accumulator26.1.4 fp16 versus bf16: the exponent–mantissa bargain26.1.5 int8 quantization for neural inference26.1.6 When float is non-negotiable26.2 Algorithmic speedups26.2.1 Separability — a 2-D pass for the price of two 1-D passes26.2.2 Recursive / IIR filters — a running state, cost independent of radius26.2.3 Integral images / summed-area tables — any box sum in four lookups26.2.4 Fast median filters — a sliding histogram for a constant-time median26.2.5 Pyramids and multiscale — do the large-scale work on small images26.2.6 Discretize the range to accelerate non-linear filters26.2.7 Downsampling and edge-aware upsampling26.2.8 Where this goes next26.3 Automatic search for fast methods26.3.1 The problem is a curve, not a point26.3.2 Why search beats a hand-shrunk CNN26.3.3 Ma et al. 2022: searching structure and parameters together26.3.4 Three axes of "search instead of design"26.3.5 The broader family, and what it costs26.3.6 Where this goes next26.4 Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast26.4.1 Why moving data, not doing math, is the bottleneck26.4.2 The memory hierarchy and locality26.4.3 The roofline: is my kernel compute- or memory-bound?26.4.4 The forms of parallelism26.4.5 What it takes to leverage a modern machine26.4.6 Why photography is hard for the machine26.4.7 Where this goes next26.5 Hardware backends: GPU, NPU, DSP26.5.1 GPU — the data-parallel workhorse26.5.2 The neural accelerator: NPU and TPU26.5.3 DSP — the real-time control loop26.5.4 ISP, FPGA, and ASIC: the fixed-function end26.5.5 The heterogeneous SoC, and the spectrum to carry away26.5.6 On-device ML runtimes, and why the work stays on the phone26.6 Halide: Decoupling Algorithms from Schedules26.6.1 What it means to separate the algorithm from the schedule26.6.2 Why image pipelines are uniquely hard to optimize26.6.3 The split was always there: done by hand26.6.4 The scheduling space26.6.5 Auto-scheduling — letting the compiler search26.6.6 Results, impact, and reach26.6.7 Gradient Halide — differentiating the pipeline26.6.8 Where this goes next26.7 Halide programming26.7.1 The three nouns: `Func`, `Var`, `Expr`26.7.2 A first image pipeline: brighten, then blur26.7.3 Reductions: `RDom`, sums, and histograms26.7.4 The default schedule, and seeing the loops26.7.5 Scheduling the loops within a stage: `reorder`, `split`, `tile`, `vectorize`, `unroll`, `parallel`, `fuse`26.7.6 The heart of it: producer–consumer granularity (`compute_at`, `store_at`)26.7.7 The schedule ladder, with numbers: ten times faster from one line26.7.8 Boundaries and bounds26.7.9 Same algorithm, new machine: the GPU26.7.10 Letting the compiler schedule: the auto-scheduler26.7.11 The development loop: correctness, measurement, and benchmarking hygiene26.8 Efficient neural network inference26.8.1 Quantization: fewer bits per weight26.8.2 Pruning: fewer weights26.8.3 Knowledge distillation: a small student, a big teacher26.8.4 Low-rank and tensor factorization26.8.5 Efficient architectures: cheapness designed in26.8.6 Neural architecture search: let the machine design it26.8.7 Hardware-aware deployment and the roofline26.8.8 Where this leaves the part27 CONCLUSIONS, DISCUSSION
27.1 Recap in contextoutline 27.1.1 Modern phones, multiple apertures, pano, HDR+27.1.2 Recap: a modern mirrorless camera27.1.3 Recap: a modern cell phone multi camera27.1.4 Lightroom27.1.5 Photoshop27.2 Why phones are so good (at photography)27.2.1 Computation beats glass27.2.2 The whole pipeline is co-designed27.2.3 Machine learning and data at scale27.2.4 Many small cameras for one big one27.2.5 The human and system advantages27.2.6 The hard caveats: physics still wins where it must27.2.7 The throughline28 BACK MATTER
28.1 Bibliography 28.2 Glossary 28.3 Acronyms 28.4 Term index 29 APPENDICES
29.1 Refreshers29.1.1 Linear algebra29.1.2 Calculus: derivatives, gradients, integrals29.1.3 Optimization and regression29.1.4 Probability and information theory29.1.5 Machine learning and deep learning29.1.6 Programming: Python, C++, and PyTorch29.2 Problem Set 0 — Environment and C++ basics29.2.1 Summary29.2.2 Installation and Environment Setup29.2.3 C++29.2.4 Submission29.3 Problem Set 1 — Image class, point operations, and color29.3.1 Summary29.3.2 The Image Class29.3.3 Brightness and Contrast29.3.4 More Image Class Methods29.3.5 Colorspaces29.3.6 Spanish Castle Illusion29.3.7 White Balance29.4 Problem Set 2 — Convolution and the bilateral filter29.4.1 Summary29.4.2 Smart Accessor29.4.3 Blurring29.4.4 Denoising using Bilateral Filtering29.4.5 Extra credit29.4.6 Submission29.5 Problem Set 3 — Denoising and demosaicking29.5.1 Summary29.5.2 Denoising from a sequence of images29.5.3 Demosaicing29.5.4 Edge-based green29.5.5 Red and blue based on green29.5.6 6.865 only (or 5% Extra Credit): Sergey Prokudin-Gorsky29.5.7 Extra credit (maximum of 10%)29.6 Problem Set 4 — High dynamic range29.6.1 Summary29.6.2 HDR merging29.6.3 Tone mapping29.6.4 Extra credit (10% max)29.7 Problem Set 5 — Resampling, warping, and morphing29.7.1 Summary29.7.2 Resampling29.7.3 Warping and morphing29.7.4 Extra credit29.8 Problem Set 6 — Homographies and manual panoramas29.8.1 Summary29.8.2 Class Morph29.8.3 Homogeneous Coordinates29.8.4 Linear Algebra29.8.5 Warp and Image with a Homography29.8.6 Compute Homography from 4 Pairs of Points29.8.7 Bounding boxes29.8.8 Extra Credit (up to 10% total)29.9 Problem Set 7 — Automatic panoramas29.9.1 Summary29.9.2 Previous Problem Set Code29.9.3 Class Morph29.9.4 Harris Corner Detection29.9.5 Descriptor and correspondences29.9.6 RANSAC29.9.7 Automatic panorama stitching29.9.8 Blending29.9.9 Mini planet29.9.10 6.8370: Stitch N Images (6.8371: Extra Credit 5%)29.9.11 Make your own panorama29.9.12 Extra credits (10% max)29.10 Problem Set 8 — Non-photorealistic rendering29.10.1 Summary29.10.2 Paintbrush splatting29.10.3 Painterly rendering29.10.4 Oriented painterly rendering29.10.5 Your image29.10.6 Paper Review (6.865 only)29.10.7 Extra credits29.11 Problem Set 9 — Make-your-own, video, and ethics29.11.1 Summary29.11.2 Make Your Own Assignment29.11.3 Ethical issues in computational photography29.11.4 Assignment Lists29.12 EXIF and image metadata29.12.1 What EXIF is29.12.2 The fields, grouped by what they describe29.12.3 How far to trust it29.12.4 Privacy: the metadata that follows the picture29.12.5 Reading and writing EXIF29.13 DNG: the Digital Negative29.13.1 What DNG is, and the problem it solves29.13.2 Inside the container29.13.3 What the raw payload looks like: mosaic vs linear29.13.4 The color recipe: matrices, profiles, and white balance29.13.5 Opcodes: corrections the decoder must apply29.13.6 Compression, and embedding the original29.13.7 Where you meet DNG: adoption and relatives29.13.8 Trade-offs, and DNG's relation to EXIF29.14 Rendering a raw DNG29.14.1 The pipeline, step by step29.14.2 Where Lightroom's "look" actually comes from29.14.3 What we would ask an Adobe engineer to check29.14.4 Two kinds of DNG, and the special cases29.14.5 Easy mistakes (most of which we made)29.15 Datasets29.15.1 Classification and features29.15.2 Super-resolution29.15.3 Deblurring and restoration29.15.4 Denoising29.15.5 HDR and tone mapping29.15.6 Retouching and enhancement29.15.7 Depth and motion29.15.8 Light fields29.15.9 Color and white balance29.15.10 Faces29.15.11 Inpainting, segmentation, and matting29.15.12 Image quality29.16 A camera-feature wish list29.16.1 Exposure, ISO, and dynamic range29.16.2 Bracketing more than exposure29.16.3 Focus and depth of field29.16.4 Computational raw and the sensor29.16.5 Motion data, metadata, and workflow29.16.6 Panorama and multi-shot29.16.7 The interface and the ecosystem29.17 How this book was created29.17.1 Two documents, not one29.17.2 Compiling a section29.17.3 Figures as code29.17.4 Generative imagery: cover art and 3D29.17.5 Verification and review29.17.6 Keeping a long book coherent29.17.7 The toolchain29.17.8 What the machine did, and what it did not29.17.9 Who wrote what: a per-part estimate29.18 The course tutor: a local, book-grounded AI teaching assistant29.18.1 What it is, and what it is for29.18.2 Local-first29.18.3 Grounded in the book: retrieval-augmented generation29.18.4 It links, it shows equations, it shows figures29.18.5 Two front-ends, one core29.18.6 What the instructor sees29.18.7 Privacy and candor29.19 The semi-automatic grading system29.19.1 The shape (to be confirmed)29.19.2 Automatic versus human (to be confirmed)29.19.3 To be filled in (from the instructor)29.20 Under the hood: prompts, patterns, and verifiers29.20.1 Prompt patterns that made it work29.20.2 The verifier suite29.20.3 Why this is the interesting part29.21 Reading a Lytro light field29.21.1 The container: the LFP/LFR format29.21.2 The pipeline, step by step29.21.3 From hexagonal lenslets to a uniform grid29.21.4 Why a white image29.21.5 Open questions and where we approximate29.21.6 Easy mistakes (most of which we made)29.22 File conversion tools29.22.1 JPG → PNG, without an alpha channel29.22.2 DNG → a linear PNG29.22.3 A Lytro capture → a Stanford-style light-field archive29.22.4 How they run in the browser29.22.5 Easy mistakes29.23 The interactive figures — a prompt-by-prompt making-of29.23.1 Summary29.23.2 Lorentz resonance — `fig-lorentz-resonance` (Figure 2.1.8)29.23.3 Rainbow droplet — `fig-rainbow-droplet` (Figure 2.1.14)29.23.4 Diffraction wave simulation — `fig-diffraction-wave-sim` (Figure 2.1.29)29.23.5 Exposure-triangle simulator — `fig-exposure-triangle-sim` (Figure 2.11.5)29.23.6 Repeated quantization — `fig-repeated-quantization` (Figure 3.1.4)29.23.7 JPEG generation loss — `fig-jpeg-generation-loss` (Figure 3.17.9)29.23.8 Exposure round-trip (JPEG) — `fig-exposure-jpeg-roundtrip` (Figure 3.17.10)29.23.9 1-D sampling pipeline — `fig-sampling-1d-demo` (Figure 3.10.4)29.23.10 2-D sampling pipeline — `fig-sampling-2d-pipeline` (Figure 3.10.6)29.23.11 Mitchell–Netravali bicubic (B,C) — `fig-bicubic-bc` (Figure 3.11.13)29.23.12 Rotation resample challenge — `fig-rotation-resample-interactive` (Figure 3.11.18)29.23.13 Mini-Lightroom — `fig-mini-lightroom` (Figure 3.18.4)29.23.14 Poisson blending — `fig-poisson-blend` (Figure 5.1.5)29.23.15 Bilateral grid (3-D) — `fig-bilateral-grid-3d` (Figure 5.2.15)29.23.16 Perspective montage — `fig-perspective-montage` (Figure 6.6.13)29.23.17 Beier–Neely morph — `fig-beier-neely-demo` (Figure 6.3.4)29.23.18 Live face landmarks — `fig-face-landmarks-live` (Figure 7.10.2)29.23.19 Lens optimizer — `fig-lens-optimizer-demo` (Figure 9.4.4)29.23.20 Full auto-panorama — `fig-pano-stitch` (Figure 10.7.9)29.23.21 CLIP-IQA curation — `fig-clip-iqa` (Figure 11.3.2)29.23.22 Photobio time-lapse — `fig-photobio-demo` (Figure 11.7.2)29.23.23 Refocus shift geometry — `fig-refocus-shift-geometry` (Figure 13.4.2)29.23.24 Chroma key — `fig-chroma-key` (Figure 8.11.7)29.23.25 Portrait-lighting simulator — `fig-portrait-lighting-sim` (Figure 2.11.26)29.23.26 Aberration explorer — `fig-aberration-explorer` (Figure 9.1.6)29.24 Interactive demo index29.24.1 Sharing, linking, and embedding a demo29.24.2 Introduction29.24.3 Fundamentals — light, optics, sensors, color29.24.4 Basic image processing and the ISP29.24.5 Computational tools — machine learning and diffusion29.24.6 Edges matter — gradient-domain and edge-preserving29.24.7 Warping and morphing29.24.8 Matching pixels across space and time29.24.9 Single-image computational photography29.24.10 Optics, lenses, and aberration correction29.24.11 Multiple-exposure imaging — HDR and panoramas29.24.12 Many images and photo collections29.24.13 Video29.24.14 Light fields and plenoptic cameras29.24.15 3-D and depth29.24.16 Appendices and end matter29.24.17 Not yet placed