INTRO From Digital to Computational Photography Computation is the new optics The goals of computational photography What makes a technique useful A field in transition: from hand designed algorithms to learning to generative AI Does photography still matter in the age of generative AI?A field at a crossroads It's not just about photography How to read this book Machine-readable by design How this book was made What programming language for computational photography?On the desktop: Python, C++, and Halide In the browser: JavaScript and the web stack On the phone: Android and iOS Libraries Vibe coding Problem sets FUNDAMENTALS OF IMAGING Light and physics Light: rays, waves, and the spectrum How light is created Light power and brightness Summing coherent vs. incoherent light Reflection, refraction, and what happens at a surface Polarization The color of objects: illumination times reflectance The BRDF and the look of materials Illuminants Radiometry: radiance, irradiance, exposure, and falloff Global illumination Wave effects, diffraction, and the diffraction limit Perceptual color and trichromatic vision Visual system anatomy:Color Opponent process and the multistage model So what are the primary colors?Human Vision Light adaptation Lightness constancy Color constancy Contrast Spatial vision Temporal vision Attention and eye movements The visual system makes things up Animal eyes The optics: many ways to form an image The color vision: opsins remixed Measuring and encoding color analysis vs synthesis and non-orthogonality Measuring color Chromaticity diagram Linear vs Gamma vs. log encoding RGB color spaces Non-linear perceptually-uniform color space HSV, HSL, and cylindrical color spaces Reproducing color Skin tones Pinhole image formation and linear perspective Pinhole imaging and the perspective projection Homogeneous coordinates Camera in a general configuration Intrinsics, extrinsics, and what cropping really does What perspective preserves, and what it destroys Wide-angle distortion: spheres bulge and faces stretch at the edges Photography with focal length: framing, magnification, and compression Depth, ray length, and unprojection Lens image formation Single lenses: refraction and Snell's law Thin lens optics Aperture and the f-number Image measurements as integrals The plenoptic function The pixel integral Exposure values (EV) and stops Splitting the integral Depth of field The circle of confusion Depth of field versus depth of focus Choosing the maximum circle of confusion The double cone, and the near and far limits Hyperfocal distance The surprising invariance Background blur beyond the focus plane Defocus is not a blur of the image Sensor size scales depth of field Motion blur Blur from subject motion Blur from camera shake How shift-invariant is camera-shake blur?Which dominates: translation or rotation?The hand-holding rule of thumb Shake from the shutter and mirror Panning: fighting subject motion by moving the camera Freezing motion with light Sensors: photosites, CCD vs CMOS The photoelectric effect Photon to number: the photosite Spectral sensitivity: matching the eye, catching photons, and white balance Beyond the visible spectrum: near-infrared, thermal, and ultraviolet Analog-to-digital conversion CCD versus CMOS Time integration: the exposure Shutters A taste of modern sensor tricks The impact of sensor size How far sensors have come Sensing color: multiplexing strategies Multiplexing in time Multiplexing in space: the color filter array Multiplexing across separate sensors: the beam-splitter Multiplexing per pixel by dispersion: color routers and nano-prisms Multiplexing per pixel by tunable absorbers: quantum dots Multiplexing in depth: stacked photodiodes Hybrid strategies Beyond trichromatic capture: full spectrum and multispectral Noise, signal-to-noise ratio and dynamic range Noise sources Noise variance is affine in pixel brightness The algebra of noise Signal-to-noise ratio: it's about ratios Dynamic range Regimes, and how close we are to the limits Imaging as a linear system Linear systems from a billboard to an image Linear systems from ray space (light fields) to an image Space–time: imaging across time When linearity breaks down Where this is going: invertibility Imaging as an inverse problem Linear algebra: how hard is this problem?Priors and the manifold of natural images Probability: the Bayesian view Optimization, inference, loss When you can design the operator — and a teaser of information theory Recap: what makes imaging hard, and where it's going Harder inverse problems: factorization Limitations of the medium Compensation, accentuation, conflict The picture is flat: depth and its cues The picture is static: time and motion One viewpoint, and a finite frame The contrast is limited: dynamic range and gamut Resolution: the limitation we are overcoming Displays The range of displays Display technology HDR displays and dual modulation The film movie projector Printing Why the display characteristics matter Distance, resolution and acuity Sharpening for the display: size and distance Light level viewing conditions Robustness of perspective to the viewer's viewpoint Gamut and gamut mapping Color management, ICC, and industry standards PHOTOGRAPHY Exposure Basic photography: exposure settings — shutter, aperture, and ISO Metering: from printed guides to the sensor f-stops versus T-stops From exposure to photometry Exposure modes, UI, and auto-ISO Lenses Lenses and focal length Aperture and the diaphragm Focusing the lens Image stabilization Keeping the lens clean Lens filters: polarizers, ND, and graduated ND Focus, autofocus, and depth of field Manual-focus aids: making the focal plane visible Cameras UI and display How the viewfinder works Anatomy of a modern full-frame interchangeable-lens camera Anatomy of cell-phone cameras Camera versus phone Types of cameras Cameras beyond photography A camera's other sensors Video Cinema lenses Illumination and the flash Natural illumination Flash Flash metering Indirect illumination and bounce flash Multiple-point lighting Traditional and Digital Darkroom The traditional darkroom The digital darkroom Programmability, or the lack thereof Photographs are usually not passive objective recordings Faithful is not the same as realistic Choosing the instant, choosing the viewpoint Speed, aperture, and focus: the photographer rules Filters and lighting: building the scene's light The darkroom: tone, dodging, and burning Make-up and retouching: authoring the subject Getting closer — systematic capture, and the biases that remain BASIC IMAGE PROCESSING AND ISP Image representation An image is an array Float vs 8-bit vs more bits Stochastic Quantization and Dithering How the array sits in memory Pixel coordinate conventions Alpha and extra channels Video and more dimensions Pixel lookups and boundaries Beyond the pixels: basic metadata and EXIF Three kinds of operation What the numbers mean What the numbers actually mean: encoding What the numbers actually mean: color spaces Developing, Testing and Debugging The workflow: build, run, look at the picture Principles Test on inputs you can verify by hand Finding good real test inputs Per-algorithm debug recipes Crashes and bounds Vibe coding: writing image code with an LLM Point operations Three kinds of operation The point-operation curve Exposure: a multiply in linear light Brightness vs. exposure Contrast: steepen about a pivot Black point, white point, and levels The general tone curve Basic color enhancement: saturation and vibrance Converting to black and white Lookup tables Video color grading Local point operations Histograms the histogram the histogram depends on the encoding space a histogram is a sampled estimate histogram equalization histogram matching constraining the slope: Ward's histogram-based tone mapping Global tone mapping The dynamic-range problem Two families: global and local Global tone mapping: one curve for everything Histogram equalization and Ward's bound Local tone mapping and the halo problem: why naive local fails Why log space pays off The analog ancestor: the Zone System HDR on every screen: the gain map Where this is going Beauty curves: the camera "look"Why scene-linear looks dull The shape of the curve Highlights: roll off to white, not to a color Saturation, and the rest of the "look"The discipline Neighborhood operations and convolution Motivation: blur and sharpen Convolution 101 The flip: where-from vs. where-to Properties: the impulse, normalization, symmetry, commutativity Nitty-gritty: finite images A blur zoo Separability Gradient and oriented filters Is imaging blur usually a shift-invariant convolution?Where this goes next Sharpening Linear sharpening Why it's called "unsharp mask"Sharpen in gamma, linear, or log?Non-linear sharpening Fourier Images as vectors in a high-dimensional space why Fourier Definition: one coefficient per wave Sines are the eigenvectors of convolution The 2-pixel example Reading an image's Fourier transform A small bestiary of transforms The fine print: two limitations of Fourier Windowing: the price of measuring a real spectrum Application preview: can we deblur, given the blur?Sampling, downsampling, and aliasing Aliasing: a high frequency in disguise Nyquist and the sampling theorem In two dimensions, aliasing has a direction Sampling in the frequency domain: spectral replicas Seeing it: sampling and pre-filtering, hands on The ideal reconstruction filter: sinc, and why it is unreachable Resampling and upsampling Domain operations: moving pixels around Start with scaling up The naive idea, and why it leaves gaps Loop over the output, use the inverse transform Nearest neighbor Linear interpolation, in 1-D first Bilinear, in 2-D The convolution perspective Better kernels: bicubic and Lanczos Upsampling vs. downsampling: scale the kernel Prefiltering when the transform isn't a clean scale Beyond Nyquist-limited reconstruction Where this goes next Pyramids and wavelets Halfway between space and frequency The idea: process an image at many resolutions The Gaussian pyramid The Laplacian pyramid Reconstruction: an exact encoder and decoder Each level is a frequency band What the bands look like on a real image Candid limitations Cousins: wavelets and steerable pyramids Pyramid blending Other applications Multiresolution as a recurring theme Image metrics Full-reference vs. no-reference why not just L2?PSNR SSIM VDP and HDR-VDP learned metrics The right metric depends on the task Denoising basics what is noise?denoising by averaging multiple shots denoising from a single image Spatial averaging and its limits The bilateral filter: averaging by affinity Denoising in the pyramid: coring Denoise color more than brightness noise estimation The limits of denoising Demosaicking Reminder: the Bayer mosaic The task: full RGB at every pixel The naive approach: interpolate each channel on its own Why naive interpolation zippers: averaging across an edge Doing better: edge-directed interpolation The harder half: red and blue, and color fringing Green-based demosaicking: interpolate the color difference Classic (non-learning) demosaicking: the general strategy Related: the optical anti-aliasing filter Quad-Bayer sensors: remosaic before demosaicking Fuji X-Trans: a larger, irregular CFA Beyond hand-tuned: joint denoising and learned demosaicking Cross-reference: other ways to sense color Where this sits in the pipeline Auto-exposure and auto white balance Auto-exposure: metering white balance and color constancy Automatic white balance The limits of white balance, and CRI File formats and compression The big picture: none, lossless, lossy Data versus metadata: EXIF PNG: the format we read from JPEG: compression by perception RAW files: before the cooking HDR formats: more than 8 bits Modern formats Other formats in passing: TIFF and GIF Recap ISP, non-destructive editing:A basic ISP Pipeline design and tuning The ISP, evolving: traditional → learned → generative Recap 2: non-destructive editingCOMPUTATIONAL TOOLSLinear Inverse Problems and RegressionBlur is linear, so deblurring is inversionImages as vectors, and the notation overloadRegression: deblurring as least squaresMatrices without forming matrices: gradient descent and conjugate gradientEfficient solversEfficient solversMatrix-free, iterative, multiscale: the situationGradient descent and conjugate gradient (recap + preconditioning)Multigrid: coarse-to-fine on a pyramidThe FFT solver: diagonalize when you canWhich solver wins — and why this is the foundation of the partInformation theoryThe basics, at a high levelCompression: the clearest applicationThe imaging system as a channelInformation-theoretic quantities as objectivesThe modern revival: learned compression and generative modelsCousins, not childrenWhere it is least relevant: aesthetics and perceptionFundamental limits of inverse problemsThe question: what can no algorithm recover?The SVD picture: null space, near-null, and the noise explosionInformation bounds: a floor no estimator beatsThe resolution–noise tradeoff, and how a prior inventsFundamental versus practical, and coded capture as the escapeMachine learningThe framing: learned operators replace hand-designed onesThe data story: synthetic data, noise models, datasetsDeep learningLow-level learned operators (pixel-to-pixel)Mid- and high-level learned predictorsGenerative models: image-to-image translationLearned perceptual metrics and lossesGenerative AI and diffusionThe framing: generation is learning and sampling a prior $p(x)$Diffusion: generation as iterated denoisingConditioning: text, images, and controlPosterior sampling: generative priors for inverse problemsOther generative families, in briefCaveats and ethicsImage priorsA catalog of priors: from smoothness to learnedShift and scale invarianceAnisotropy: sparsity in a single directionLinear color modelsDenoising as a universal prior: Plug-and-Play and REDDiffusion is iterated denoising (the continuous limit)Score-based models, the thin manifold, and imperfect priorsEDGES MATTERBilateral filteringMotivation: local tone-mapping haloesBilateral filterCross / joint bilateralBilateral gridBilateral-grid learning (HDRnet)Non-local meansEdge-preserving optimization — colorizationThe shift: filtering becomes optimizationColorization: the canonical demonstrationWriting down the energySolving it: a sparse linear system we already ownOne affinity, two faces: the matting Laplacian connectionLocal Laplacian filtersThe wish, and why the bilateral leaves a haloWhat a Laplacian pyramid hands youThe mechanism: a remapping recomputed per output pixelOne curve, three edits: enhancement, tone mapping, inverse tone mappingContrast with the bilateral base/detail splitThe cost, and the fast versionMatting LaplacianThe matting problem: compositing, and why it is under-determinedThe color-line assumption: α becomes linear in colorClosed-form matting: the matting LaplacianSpectral matting: eigenvectors of the matting LaplacianAn alternative matting LaplacianBeyond matting: spatially-varying white balanceGuided image filteringThe key idea: the output is a local line in the guideFitting the line, and the one knobWhy it is fast: O(N) regardless of window sizeNo gradient reversalReading the affinity back outWhere it is usedLocally adaptive regression kernel (LARK)Change the verb: from averaging to fittingSteering the kernel to the local structureThe regression flavor of the affinityLARK as a structure descriptorPoisson image editingEdit gradients, not pixelsSeamless cloning: the headline applicationReconstruction = solving the Poisson equationPoisson vs. pyramid blendingAdvanced techniquesSeam optimizationFind the non-edges: least-noticeable cutsIntelligent scissors / live-wire: the cut as a least-cost pathSnakes (active contours): the continuous cousinGraph cut: globally optimal boundaries by min-cut / max-flowGrabCut: interactive cutout from a single rectangleNormalized cuts: the spectral relaxationSeam carving: content-aware resize by DPTime-lapse via DP: a seam through timeVideo textures: looping a seam in timeGraphCut textures & photomontage: cut, then blendWhere this connects: MRFs and beyondRecap: which edge-aware technique when?The three relationships to edgesPros, cons, and cost: when to reach for eachA worked chooserWARPING, MORPHING, AND LAGRANGIAN APPROACHESWarpingWarping is a domain transform: move where, not whatForward vs inverse warping: why output-driven winsResampling: a quick reminder (developed in BASIC)Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladderSpecifying the warp II: free-form warps from sparse correspondencesLiquify: the warp with a paintbrush bolted onAntialiasing for complex transformsWhen the footprint stops being a square: the warp JacobianEWA: the elliptical weighted averageFeline and anisotropic MIP probing: EWA on a GPU budgetOther footprint integrators: summed-area tables and ripmapsFootprint estimation, clamping, and when to reach for itMorphingWhy a cross-dissolve isn't enough: the ghosting motivationTwo interpolations: domain (shape) and range (color)The morphing recipe (combine both)Field morphing: the Beier–Neely line-pair warpMesh / triangulation morphing (the alternative warp)View morphing: the geometrically-correct in-between of two viewsRecap and significanceMorphable modelsStep 1: dense correspondence, the precondition for averaging shapesStep 2: Procrustes alignment, factor out pose so PCA sees shapeStep 3: PCA, a mean and a basis of eigen-deformationsStep 4: shape and appearance are separate basesStep 5: fitting a new photo, analysis-by-synthesisStep 6: once fitted, edit by moving in the spaceStep 7: from linear PCA to neural priorsRecap and significanceShape-preserving warpingWhy plain interpolation shears: the rigidity gapAs-rigid-as-possible (ARAP) manipulationLinear blend skinning and why the weights decide everythingBounded biharmonic weights (BBW)ARAP interpolation: rigid-as-possible morphing between two posesMoving least squares: the simpler cousinRecap and significancePerspective distortion and its correctionKeystoning is projection, not a lens flawThe fix is a homography: re-render the façade fronto-parallelThe optical alternative at capture: tilt-shift / ScheimpflugCropping wide-angle photographs: recenter with a homography, don't just cropA different perspective distortion: wide-angle portraits, and a content-aware fixBeyond faces: correcting arbitrary objects (MaDCoW)A montage has no single viewpoint: a family in a boxRecomposing perspective after capture: computational zoomThe catch: resampling cost, and "only a plane rectifies exactly"Where this sits: one map, then transportMATCHING PIXELS AND HUMANS ACROSS SPACE AND TIMEBrute forceWhy you must align firstBrute-force translational alignment (SSD / NCC)Phase correlation: the whole shift from one FFTCoarse-to-fine alignment on a pyramidSub-pixel matchingRefining the cost: fit a parabolaRefining in the frequency domain: phase correlationRefining by gradient: the Lucas–Kanade stepWhat breaks it: peak-locking, texture, and noiseOne tool, everywhere downstreamSparse matchingWhere to look: corners, and the structure tensorInvariance: surviving scale and rotationDescribing a neighborhood: SIFT and its zooMatching: nearest neighbor in descriptor spaceFeature trackingTracking vs dense flow: sparse-but-long vs dense-but-shortKLT = Lucas–Kanade, per feature, iterated over timeGood features to track = where the structure tensor is well-conditionedWhat breaks long-term tracking: drift, appearance change, occlusion, and re-detectionModern point trackers (briefly)An application: synthetic motion blur from a trackRobustness: the ratio test and RANSACThe ratio test: reject ambiguous matches before fittingRANSAC: fit from a minimal random sample, score by consensusVariants and degeneracyDeep learning approaches to sparse matchingLearned local features: SIFT's job, done by a networkLearned matching — reason about the whole set at onceThe 3-D-aware turn: pointmaps subsume matchingFast matchingApproximate nearest neighbors for sparse descriptorsPatchMatch: randomized dense correspondence by propagationFast high-dimensional matching by random projectionOptical flowWhat optical flow is: and whether it is even well-definedBrightness constancy and the optical-flow constraintThe aperture problemLucas–Kanade: local constant-flow least squares (and the structure tensor)Horn–Schunck: global smoothness regularizationLarge motion: coarse-to-fine warpingLearned flow: RAFT (neuralize the classical pipeline)Deep learning approaches to optical flowThe unrolling principle: neuralize the classical solverCost volumes and warping inside the netRAFT and the recurrent updateFace trackingDetecting the faceLandmarks: pinning down the featuresTracking across timeLifting to 3-D: the morphable modelRecognition, and the dark sideWhich library to useBody pose estimationTop-down vs bottom-upOn-device, real timeLifting to 3-D: parametric bodiesWhich library to useSINGLE IMAGE COMPUTATIONAL PHOTOGRAPHYDenoisingA reminder: bilateral filtering and wavelet/pyramid shrinkageBM3D: group similar patches and filter them togetherLearning to denoiseSuper-resolutionWhat problem super-resolution solves (and why it's ill-posed)Scenarios: single-image, burst, and hybrid space–timeReconstruction vs hallucination: measured detail vs invented detailDemosaicking and joint reconstructionThe pipeline problemFlexISP: one energy, one priorLearning the joint prior: the founding formulationModel-based and unrolled networksLearning where the hard cases areCo-designing the mosaic: a learnable sensorThe end-to-end learned ISPBetter backbones: CNN → transformer → diffusion priorExotic mosaics: learned remosaickingWhere do the labels come from?Correcting chromatic aberration jointlyRecapNon-blind deblurringDeblurring in the presence of noise: why naive inversion failsThe Wiener filter — the regularized, noise-aware inverseSparse gradients: the prior that keeps edges sharpBlind deblurringBlind deblurring: estimating the kernel and the imageA more realistic blur model: spatially-varying (camera-shake) blurEngineering the aperture: depth and all-focus from a coded maskDehazingDehazing as a prior-driven inverse problemDifferentiable image pipelines and algorithm optimization (Halide)Mixed-lighting white balanceWhy a global gain must failEstimating the per-pixel mixture: an under-determined inverse problemCorrecting each region for its own lightWhat came afterInpainting, texture synthesis, and object removalInpainting as filling unmeasured pixels — the spectrum of priorsPDE / diffusion-based inpaintingTexture synthesis (Efros–Leung; Efros–Freeman quilting)Exemplar inpainting: clone, healing brush, and object removal (Criminisi)Data-driven scene completion (Hays & Efros)Deep inpainting (context encoders → partial/gated conv → diffusion)Highlight / specular recoveryEpitomes — a compact patch modelPatch matchThe nearest-neighbor field, and why exhaustive search is the bottleneckPatchMatch — randomized correspondence Barnes et al. 2009Applications: hole filling, retargeting, reshuffleShift-Map image editing Pritch et al. 2009 — editing as graph-cut labelingColorizationOne channel in, three channels outA spectrum of priors: scribbles, references, and learned modelsThe multimodality trap: why naïve colorization goes muddyClosing the loop: learned priors with a human's hintsPlausible is not correctWhere it sitsCompositing, segmentation and mattingCompositing and alpha channelsSegmentation: cutting the object outThe fundamental matting equationBlue/green-screen matting and chroma keyingTraditional matting approachesDeep-learning mattingGenerative mattingHarmonization and where the blends liveOptical effects beyond alphaIllumination related effects in a single imageIntrinsic images: the unifying frame ($I = R\cdot S$)Multiple-light / spatially-varying white balanceReflection removal — pulling apart a transmission and a glass reflectionShadow detection and shadow removalSpecular-highlight removal / "fake polarization" (the dichromatic model)Tone MappingGlobal vs local, re-hashedA smarter global curve: histogram adjustmentA taxonomy of local methodsThe darkroom ancestor: dodge & burn and the Zone SystemStyle transferClassical style transfer: patches and statisticsNeural style and feed-forward stylizationStyle transfer as image-to-image translationNon-photorealistic renderingWhat NPR is for, and the one ideaStroke-based / painterly rendering (and the brush p-set)Edge-preserving abstraction: bilateral + Difference-of-GaussiansExample-based stylization and the bridge to neural styleRegion-based stylization: stained glass, low-poly, mosaicsArtistic screening and halftoningCOMPOUND LENSES, AND ABERRATION CORRECTIONAberrations and optical challengesTaxonomy of challengesSpherical aberrationComaAstigmatismField curvatureChromatic aberrationRadial distortionWave effects and diffractionVignettingFlare and coatingAberrations correctionThe two families of cureCorrection in glassComputational correctionRadial distortion correctionMeasuring lens qualityThe Modulation Transfer Function (MTF)Measuring MTF in practiceSpot diagrams and the PSFMTF through focus and depth of focusField-dependent MTF: sagittal vs tangentialColor, geometry, and illumination measurementsScalar quality summariesLens optimizationThe high-level idea: design as optimizationThe forward model: ray-tracing and spot diagramsFrom hand calculation to softwareTradeoffs: the design is always a compromiseTolerancing: from the nominal design to a manufacturable oneA short bestiary of classic designsThe lens as a system: cardinal points, pupils, f-number, T-stopScaling laws in opticsLohmann's scaling laws: why a good lens is heavyThe gigapixel barrier for a single lensThe escape: monocentric multiscale opticsCapture everything, crop later: the spatial cousin of the light fieldSpecial opticsTilt-shift and the Scheimpflug principleFisheye and non-rectilinear projectionMirrors: catadioptric and reflecting systemsPeriscope / folded-lens design (smartphone telephoto)Anamorphic opticsStereo (3D) lensesMacro, microscope objectives, and telescopesTeleconvertersFrom shaped glass to thin structures: Fresnel, diffractive, GRIN, metalensesFocusFocus mechanicsFocusing a compound lens: beyond unit focusingFocus actuatorsFocus stacking, macro, and focusing railsAutofocusContrast-detection AFPhase-detection AF (the split-pupil / stereo trick)On-sensor PDAF and dual-pixel AFDepth from focus / defocus, and learned subject AFWhere to focus: saliency, faces, and eyesFocusing in astrophotographyBokeh, focus stacking, and depth-of-field controlRecap: the geometry of focus (pointer, not re-derivation)The bokeh look: shape and structure of the blurExtending DoF: focus stackingControlling and faking DoFFake (synthetic) depth of fieldWhy phones must fake itWhere the depth (or subject) comes fromFrom depth to blur — the thin-lens circleRealistic bokeh — why a Gaussian looks fakeOcclusion-aware compositing and mattingFailure modesGlare suppressionWhere stray light comes from: flare, ghosting, veiling glareHardware suppressionComputational deflare and glare deconvolutionOptical stabilizationThe problem: hand-shake and the blur budgetOptical stabilization: lens-shift vs. sensor-shift (IBIS)Digital / electronic stabilization and the computational alternativesThe eye as an optical instrument: vision and its correctionThe eye as a cameraRefractive errors: the eye out of focusCorrecting visionPresbyopia and the bifocal problem: from bifocals to AF glassesMULTIPLE EXPOSURE IMAGINGDenoising by averagingWhy averaging works — the $1/\sqrt N$ derivationWhen the plain mean is wrong: robust combinationCalibration frames: what averaging can't fixHandheld low light: the phone in your pocketDeep-sky astrophotography: averaging at the extremeHDR mergingThe HDR challengeData capture: how to vary the exposureCurve calibrationCombining exposuresOptimizing the capture and mergeIn-sensor HDR: dynamic range without a bracketApplication to cell phones: HDR+ and burst imagingWhy a phone shoots a burst, and why it underexposesThe HDR+ pipeline: align and robust-merge in rawFrom burst HDR to burst super-resolutionMultiframe or burst super-resolutionThe reconstruction principle: many coarse grids make one fine gridSub-pixel registration is the whole gameFusion: from scattered samples to a sharp imageAccidental versus deliberate offsets: hand tremor and pixel-shift sensorsThe learned eraWhere it runs outManual panorama stitching from multiple viewsThe scenario, and the one rule: rotate, don't translateRefresher: pinhole projection is "divide by depth"Why you don't need 3D: depth cancels for a pure rotationHomographies and homogeneous coordinatesSolving for $H$ from correspondencesWarping and assembling the panoramaAnother application: document flattening and mergingAutomatic panorama stitching from multiple views and feature matchingWhy not brute force, and the two sub-problemsThe feature pipeline, recalled from Part 7RANSAC for a homographyBlendingWhy a hard seam is visible — the photometric mismatchFeathering / alpha blending — and why it ghostsTwo-scale blending — the simple split (the pset method)Multiband / Laplacian-pyramid blending — a transition per bandPoisson / gradient-domain blending — paste gradients, solve for valuesSeam optimization — route the seam instead of fading itThe complete pipeline, end to endBells and whistlesOther projectionsBundle adjustmentMovement and parallax handlingContinuous panoramas (e.g. on cell phones)Incremental registration of a video streamMosaicking a moving strip (and why a central strip)Rolling shutter and exposure driftFocal stacks and depth of field extensionWhy limited depth of field is the problemA simple algorithm: sharpness = local high-frequency energy, then argmaxThe more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)Capturing the stack — hardware, and the magnification trapHyperspectral imaging, color wheelsWhy three numbers aren't enough — RGB as a 3-sample projectionBuilding the spectral stack — filter wheels, tunable filters, pushbroom, snapshotWhat it's for — material ID, agriculture, art and beyondPolarization imagingWhat a camera throws away — polarization as a third axis of lightCapturing the stack — rotate a polarizer, or a polarization mosaicCombining images at different polarizationsIntrinsic images with time lapseThe split, and why one image can't do itWeiss 2001 — the median of log-gradientsWhere this chapter belongs — passive vs. active illuminationLucky imaging (planetary / lunar astro)Atmospheric seeing — why one long exposure failsShoot thousands, keep the sharpestAlign and stack the survivorsA poor man's adaptive opticsMANY IMAGES AND PHOTO COLLECTIONSPhoto MosaicsThe tiling-and-matching pipelineColor correction and avoiding repeatsMulti-scale and irregular tilingsWhy it resolves into the target at a distanceRetrievalClassic CBIR — histograms and the text-retrieval analogyDeep retrieval — learned embeddings and CLIPRetrieval at scale — approximate nearest neighborMining what makes a place distinctiveAuto curationTechnical quality — the easy rejectsAesthetics — the hard, learned partGrouping, summary, and diversityArranging the collection, not just culling itLife logging camerasThe devices and how they fireThe memory-prosthesis reframingThe big-data problem passive capture createsPrivacy and ethicsInpainting Using Millions of PhotographsWhy self-similar inpainting isn't enoughScene completion from a huge databaseThe data is the prior — and its modern oppositePhoto tourismStructure-from-motion on internet collectionsFrom reconstruction to experienceLineagePhotobiosAlign and order the collectionLet the data fill the gapsCollection as experienceAverage ExplorerGalton's composite portraiture — and what it was forAlignment is everythingThe average of a categoryAn artistic lineageAverageExplorer: averaging made interactiveWhat an average is, and what it is good forPix 2 GPSGeolocation as retrieval over a geotagged corpusThe answer is a distributionMapping the collection itselfThe learned successorsPersonalized priorsPersonalized restorationPersonalizing generative modelsThe bargain and its ethicsArtistic projects with photo collectionsStatistical collage — SalavonAnticliché cameraPareidoliaDisplaying images togetherSelectionLayoutColor and coherenceThemesVIDEOMotion blur, temporal sampling, and resamplingA frame is an integral over time → motion blurTime is sampled → temporal aliasing, the wagon-wheel effectMotion blur is the temporal prefilter: the two are one tradeoffResampling in time: frame-rate conversionLagrangian vs Eulerian: the organizing distinction for the partVideo compression and motion compensationWhy video compresses far better than still × N: temporal redundancyMotion-compensated prediction: the core trickI, P, and B frames; GOP structureWhy this is "optical flow on a budget"Modern codecs in one breathVideo editingNon-linear editing: the timeline metaphorSummarization: keyframes, fast-forward, and highlightsFun temporal filters: reduce-over-timeTranscript-based editingIn-betweening: synthesizing the frames an edit needsCoda: storyboards, interviews, and where this part landsFrame interpolation and slow-motion synthesisWhy interpolate: faking slow-motion and up-converting frame rateInterpolation = morphing between adjacent framesFlow-based interpolation: warp both frames to the midpoint and blendLearned synthesis: Super SloMo and FILMHybrid low/high resolution and frame-rate imagingThe space–time bandwidth trade-offThe hybrid two-camera architectureMotion from the fast streamApplications: deblur, space–time super-resolution, video from stillsModern descendantsVideo stabilization and rolling-shutter correctionWhat stabilization is: a camera-path signal to be smoothedStage 1: estimating the camera trajectoryStage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)Stage 3: re-rendering and the stabilization↔crop tradeoffRolling-shutter correction: per-row pose and rectificationTime-lapse photographyCapturing a time-lapse: interval, shutter, and day-to-night transitionsDeflickering: stabilizing exposure, white balance, and lightHyperlapse: stabilizing a time-lapse that also movesMining time-lapses from the internetFactoring a time-lapse: reflectance, illumination, and intrinsic imagesVideo texturesFinding good transitionsPlaying itRelatives and descendantsLIGHT FIELDS AND PLENOPTIC CAMERASLight fields 101Capture rays, not pixelsThe plenoptic function, reduced to four dimensionsRay and point are dualReading the 4-D structure through its 2-D slicesRendering a new view by looking up raysLight fields vs. plenoptic and radianceLight field camerasIntegral photography: Lippmann's fly's-eye plate (1908)The plenoptic camera: a microlens array on the sensorLytro: the consumer plenoptic cameraCommercial light-field cameras beyond LytroCamera arrays: a grid of full camerasThe spatial↔angular tradeoffTwo strategies, and why arrays and microlenses are dualOther light field acquisition setupsOne camera on a gantry: sample the aperture in timeHandheld, unstructured capture: let the poses be irregularCatadioptric capture: one sensor, many viewpoints at onceCoded aperture in time: sweep the pupil itselfThe everything-else, mappedRefocusing and synthetic apertureReconstructing a photo: it is all about which rays you sumDigital refocusing is shift-and-addA focal stack from one capture, and an all-in-focus imageFourier-slice photography: the fast versionReading refocus and depth off an epipolar sliceSynthetic aperture: an aperture the size of a roomAberration correction in light fieldsAn aberration is misrouted raysRe-routing each ray to the ideal-lens positionThe trade: ray bookkeeping instead of glass, and only what you sampledLight field aliasing and 4D Fourier analysisThe light field is a sampled signal, and its samples are viewpointsDepth is slope is spectral orientationThe bowtie: a spectrum shaped by the scene's depth rangeHow densely must you sample? The plenoptic-sampling boundGeometry buys back samples: the depth-vs-views tradeoffUnder-sampling looks like a ghostWhere this sitsLumigraph and shape priors for sharper light field renderingPure light-field rendering blurs because it has no shapeThe Lumigraph: reproject onto a geometry proxy, then blendUnstructured inputs: free-hand views, no grid requiredSurface light fieldsLight field microscopyThe optical setup: a microlens array at the intermediate image planeThe spatial-versus-angular trade: a coarse 3-D volumeRecovering the volume: synthetic refocusing, then 3-D deconvolutionWhy single-shot 3-D is the whole pointLight field networksNeRF: a radiance field rendered by volume integrationLight field networks: a ray straight to color, in one evaluationA family of neural light fieldsTest-time training: the network as the per-scene priorHand-off: from rays to radiance fields and generationPractical aspects of light field camerasDo you lose all that resolution?Can you do video? Is it practical? The bandwidth problemThe time dimension is high speedAdjacent frontiers, brieflySo, do I get my camera?MULTI-APERTURE IMAGINGCamera arrays: one rig, many instrumentsOne rig, four instrumentsCommercial arrays: the Light L16Where this sitsBullet timeMulti-camera phonesCOMPUTATIONAL OPTICS AND CODED IMAGINGWavefront codingWhy you cannot just deblur defocusThe fix: re-engineer the blurThe cubic phase plateThe depth-invariant PSF and a single deconvolutionWhat it costsWhere it sitsCompressive sensingSub-Nyquist: fewer measurements than unknownsTwo ingredients: incoherent measurements and sparsityRecovery by $\ell_1$: the geometry of basis pursuitWhy it works: the restricted isometry property, intuitivelyThe single-pixel cameraWhere compressive sensing pays off — and where it does notCoded apertureThe bad forward operator of a clear apertureA mask designed for a flat, zero-free spectrumOne shot, two outputs: depth and an all-in-focus imageThe design criterion: which pattern?Coded-aperture pairs: splitting the trade across two shotsHeterodyning the light field: dappled photographyWhere it sitsPhase-coded aperturesA reminder on wavefront codingFocus sweepThe lattice-focal lensWhere it sitsCode in time (phase, amplitude)Why ordinary motion blur is (almost) unrecoverableThe flutter shutter: chop the exposure into a codeThe decode: one deconvolution, a sharp moving objectAmplitude in time, and phase in timeCoded strobing, temporal multiplexing, and compressive videoMotion-invariant photographyWhere it sitsTheoretical analysis of imaging systems in the 4D light field Fourier domainThe light field's spectrum, and the one move that explains everythingEvery camera is a different slicePutting cameras on one footing: the Bayesian comparisonThe lattice-focal lens: tiling the wedgeThe upper bound — and the gap we have not closedFrom cameras to light transport: the same spectral lensWhere this leaves the partEnd-to-end optimizationThe pipeline as one differentiable graphBackpropagating into the glassWhat gets designed: a height map, not a hyperparameterA gallery of deep-optics resultsWhat it costs, and where it can go wrongWhere it sitsFourier opticsLight as a wave: amplitude, phase, and what "coherent" meansDiffraction is a Fourier transformA lens computes a Fourier transformThe pupil function is the transfer functionThe diffraction limitAberrations are pupil phaseEvery code in this part is a choice of pupilFourier ptychography: synthesizing a bigger pupilExotic / advanced opticsLensless imagingGRINMetalenses and advanced crazy optics à la Barbastathis (coherent though)Non-linear opticsOptical modulators (spatial light modulators): DMD and LCD/LCoSCOMPUTATIONAL SENSORSAssorted pixelsDual-pixel and phase-detect pixels: buying depth and focusClear, white, and other color-filter variantsPolarization pixelsSpatially varying exposure: assorting for dynamic rangeThe common threadModern sensors (quad Bayer, in-sensor HDR, and beyond)Quad Bayer, Tetracell, and nona-binningIn-sensor HDR: capturing range before the mergeDual-pixel autofocus, on the same sensorBSI and stacked sensors: compute under the pixelsThe global-versus-rolling shutter tradeBeyond: nano-prism, organic, and event pixelsOn-sensor HDRStaggered / multiple-exposure readout (DOL-HDR)Dual (and triple) conversion gain (DCG)Split-pixel: a large and a small photodiodeSpatially-varying exposure (SVE) / assorted exposuresLateral overflow integration capacitor (LOFIC)Logarithmic, self-resetting, and counting pixelsOn-sensor HDR versus multi-frame HDRDepth sensorsStereo: passive triangulationStructured light: projecting the textureLiDAR and direct time-of-flightTime of flightPassive depth from one camera, in passingThe menu, in one viewSingle-photon sensors (SPAD, avalanche, photon counting)From avalanche gain to a single-photon clickPhoton-counting arrays: zero read noise, shot-noise-limitedThe Quanta Image Sensor: a different road to one photonWhat it costs: dark counts, dead time, fill factor, data rateDoppler / velocity imagingThe Doppler shift, as a velocity sensorThe instruments: vibrometry, Doppler LiDAR, radarWhere it fitsEvent sensorsHow it works: per-pixel change detectionThe upside: microseconds, dynamic range, no blur, little dataThe downside: no picture, and a new kind of dataUses, lineage, and the contrast with single-photonSpecialized and research sensorsGeiger-mode avalanche-photodiode arrays: photon-counting laser radarDigital-pixel focal-plane arraysScientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaicsStacked and processing-in-pixel sensorsCurved focal planesBeyond the visible, and filter-array sensorsRadiation-hardened and defense focal planesThe chapter's pointExtra sensors and non-visual dataAccelerometer and gyroscope: the inertial measurement unitSound: microphones, audio-visual sync, and the visual microphoneGPS: geotagging and placeCompass and magnetometer: heading and orientationNear-infrared: the cut filter, dark flash, and NIR-assisted denoiseTemperature: dark-current compensationUltra High speed ImagingStreak cameras: sweeping time onto a spatial axisFemto-photography: a movie of light in flightCompressive ultrafast photography: a single-shot coded streakTransient imaging and looking around cornersThe bridge to direct time-of-flight and LiDARCOMPUTATIONAL ILLUMINATIONFlash photographyFlash / no flashRamesh's multiflashRemoving flash artifactsDark flash (plus Stasi version!)High-speed and stroboscopic photographyFreezing motion: the microsecond strobeThe trigger problemStroboscopic multiplicity: a sequence on one frameDigital descendants: LED strobes and high-speed camerasIllumination-based mattingThe well-posed case to beat: chroma keyThe magic prism: Disney's sodium-vapor processNear-infrared and time-multiplexed mattingFlash/no-flash matting: separation by falloffThe throughline: control the capture, not the priorSeparation of Direct and Global IlluminationThe frequency insightNayar's program: programmable, structured illuminationToward coherent separationLight domesThe reflectance field and one-light-at-a-time captureRelighting by linear combinationScaling down: tabletop LED domesScaling out: the dome taken into the wildThe found dome: the eyeAutomatic aesthetic lightingComputational bounce flashDrone lighting: flying the light into placeDual photographyLight transport as a matrixHelmholtz reciprocity: transpose the matrixThe unsettling reach: privacy and seeing the unseenCoherent imagingThe confocal principle: rejecting out-of-focus lightSeeing through scattering mediaThe part in one line3D AND DEPTHMultiple view geometryTwo views: stereo and disparityTwo-view geometry: epipolar lines, and the essential and fundamental matricesWhat "depth" means, and where it comes fromDepth is the z-coordinate, not the ray lengthWhere depth comes from: a cue-and-sensor inventoryRelative vs metric: the scale you usually don't haveMonocular depth estimation (one image → depth)Why one image cannot determine depthRelative depth, and the scale-and-shift ambiguityFrom hand-built priors to borrowed diffusion priorsWhat the maps are good forSingle-image 3-D: tour into the picture, photo pop-up, 3-D Ken BurnsThe universal recipe, and why holes are the hard partTour Into the Picture: the spidery meshAutomatic Photo Pop-up3-D Ken Burns and 3-D photos: the monocular formMulti-view 3-D reconstruction: the classic pipelineThe pipeline, stage by stageWhy it is brittle: and why SfM survives anywayStructured light scanningProjector as inverse camera: triangulation with trivial correspondenceThe coding ladder, and the frames-versus-motion tradeCalibration and failure modesPhotos → radiance fields and Gaussian splatting (NeRF, 3DGS)Two goals, one diagramInverse differentiable rendering, and the discontinuity that forced fuzzinessNeRF: a scene as a tiny neural networkDo we even need the network?3-D Gaussian Splatting: the lessons without the deep learningThe practical recipe: and what is baked inRelaxing the assumptions: NeRF in the wildFeed-forward (amortized) 3-D: skip the per-scene optimizationAmortization: pay once, reuse foreverThe line: DUSt3R, MASt3R, VGGTThe punchline, and the loop it closesThe trade, and the data dependencyRe-photographyWhy you cannot just overlayThe real-time guidance loopWhere it sitsThe landscape, and is 3-D a "fake task"?The field as a landscape, not a lineIs 3-D a "fake task"?The counterpoint, and the frontierINTEGRAL AND IMMERSIVE IMAGINGStereo glassesWheatstone's stereoscope (1838)Routing a different image to each eyeShooting and synthesizing a stereo pairVR gogglesThe three levelsThe optics: a microdisplay and a magnifier per eyeTracking, latency, and why headsets used to make people sickPassthrough, mixed reality, and where today's products sit3D displays with accommodationFour ways to deliver a focus cueThe other gaps, and the ultimate displayLenticular displaysLippmann's integral photography: the common ancestorMulti-view, and the resolution–views tradeoffLight-field telepresence: Google StarlineDisplay depth of field and antialiasingHolographyLippmann and Gabor: recording the waveOff-axis holography and the space-bandwidth wallComputational holographyRetinal projectionThe Maxwellian view: focus set by the display, not the eyeFrom the virtual retinal display to laser eyewearThe frontier: writing to individual conesREVEALING THE INVISIBLEAccidental camerasThe accidental pinhole: a window is a cameraThe accidental pinspeck: the anti-pinholeThe occluder as a crude lens, and recovery as deconvolutionCorners and doorways: an edge that resolves the hidden roomWhere else the world hides a cameraReflections in the eyeThe cornea as a catadioptric mirrorThe geometry: from a corneal pixel to a direction in the worldWhat the recovered reflection is good forEyes for relightingMotion and video magnificationThe Lagrangian precursor: track, then exaggerateEulerian video magnification: amplify the time series at each pixelWhy amplifying brightness amplifies motionPhase-based magnification: move the motion into phaseWhat it reveals: vital signs, structures, materials, modesVisual microphoneFrom sub-pixel motion to a sound waveformBandwidth: high-speed cameras and the rolling-shutter trickHow good is the copy? The object's frequency responseThe active cousins: laser vibrometry and interferometryCorner cameraThe edge as a one-dimensional apertureFrom a faint gradient to a usable signalWhat the corner can and cannot tell youActive non-line-of-sightThird-bounce geometry and time-of-flightThe hardware: photographing light in flightFrom back-projection to fast, exact inversionThe trade, stated plainlyPassive non-line-of-sightThe occluder is what makes it solvableA deconvolution where the lens is unknownOne dimension: the corner cameraTwo dimensions from a single photo: computational periscopyActive versus passive, the ledgerMm-wave, wifiWhy radio walks through wallsTime of flight, again — radar is NLOS with a longer waveFrom a radio smear to a human skeleton: the learned mapWhat it sees, and what it costs usADJACENT FIELDS AND APPLICATIONSOptical computingAstroExtreme long exposureTracking, stacking, and selection (pointers)X-rayMedicalMicroscopyMm-waveMusic, soundFluorescenceOpto-acousticUltrasoundAerial imagingComputer visionRobotics, drivingHUMAN FACTORSHuman factors and the art of photographyMake better photosTypical shooting scenariosMacro photographySpecial effect photographyFun artsy stuffPerception of artEthics of computational photographyComputational models of perceptionSpatial (and spatio-temporal) visionUser studiesAccessibility: photography by and for blind usersBlind camera — capture without a sighted operatorThe social and personal practice of photographyIMAGE FORENSICS AND AUTHENTICATIONImage ForensicsThe problem and the threat modelSensor and pipeline traces: PRNU, CFA, and noiseCompression, geometry, and metadata forensicsDeepfakes, GAN/diffusion fingerprints, and learned detectionWhy forensics is evidence, not proof — and the hand-off to provenanceAuthentication and Provenance (C2PA)From detection to attestation: trustworthy cameras and watermarkingC2PA and Content Credentials: the standardAI disclosure, watermarking, and the regulatory pushLimits, critiques, and the forensics partnershipSYSTEMSProgrammable and modular camerasImage processing librariesLightroom-style raw developersPhotoshop-style editorsNetworking and image transportPhotography programming on phonesPERFORMANCE ENGINEERING AND HALIDE8-bit and fixed-point arithmeticInteger versus fixed-point: where the binary point sitsRounding, dithering, and the banding trapSaturation and the width of the accumulatorfp16 versus bf16: the exponent–mantissa bargainint8 quantization for neural inferenceWhen float is non-negotiableAlgorithmic speedupsSeparability — a 2-D pass for the price of two 1-D passesRecursive / IIR filters — a running state, cost independent of radiusIntegral images / summed-area tables — any box sum in four lookupsFast median filters — a sliding histogram for a constant-time medianPyramids and multiscale — do the large-scale work on small imagesDiscretize the range to accelerate non-linear filtersDownsampling and edge-aware upsamplingWhere this goes nextAutomatic search for fast methodsThe problem is a curve, not a pointWhy search beats a hand-shrunk CNNMa et al. 2022: searching structure and parameters togetherThree axes of "search instead of design"The broader family, and what it costsWhere this goes nextModern CPUs: memory hierarchy, parallelism, and what it takes to go fastWhy moving data, not doing math, is the bottleneckThe memory hierarchy and localityThe roofline: is my kernel compute- or memory-bound?The forms of parallelismWhat it takes to leverage a modern machineWhy photography is hard for the machineWhere this goes nextHardware backends: GPU, NPU, DSPGPU — the data-parallel workhorseThe neural accelerator: NPU and TPUDSP — the real-time control loopISP, FPGA, and ASIC: the fixed-function endThe heterogeneous SoC, and the spectrum to carry awayOn-device ML runtimes, and why the work stays on the phoneHalide: Decoupling Algorithms from SchedulesWhat it means to separate the algorithm from the scheduleWhy image pipelines are uniquely hard to optimizeThe split was always there: done by handThe scheduling spaceAuto-scheduling — letting the compiler searchResults, impact, and reachGradient Halide — differentiating the pipelineWhere this goes nextHalide programmingThe three nouns: Func, Var, ExprA first image pipeline: brighten, then blurReductions: RDom, sums, and histogramsThe default schedule, and seeing the loopsScheduling the loops within a stage: reorder, split, tile, vectorize, unroll, parallel, fuseThe heart of it: producer–consumer granularity (compute_at, store_at)The schedule ladder, with numbers: ten times faster from one lineBoundaries and boundsSame algorithm, new machine: the GPULetting the compiler schedule: the auto-schedulerThe development loop: correctness, measurement, and benchmarking hygieneEfficient neural network inferenceQuantization: fewer bits per weightPruning: fewer weightsKnowledge distillation: a small student, a big teacherLow-rank and tensor factorizationEfficient architectures: cheapness designed inNeural architecture search: let the machine design itHardware-aware deployment and the rooflineWhere this leaves the partCONCLUSIONS, DISCUSSIONRecap in contextModern phones, multiple apertures, pano, HDR+Recap: a modern mirrorless cameraRecap: a modern cell phone multi cameraLightroomPhotoshopWhy phones are so good (at photography)Computation beats glassThe whole pipeline is co-designedMachine learning and data at scaleMany small cameras for one big oneThe human and system advantagesThe hard caveats: physics still wins where it mustThe throughlineBACK MATTERBibliographyGlossaryAcronymsTerm indexAPPENDICESRefreshersLinear algebraCalculus: derivatives, gradients, integralsOptimization and regressionProbability and information theoryMachine learning and deep learningProgramming: Python, C++, and PyTorchProblem Set 0 — Environment and C++ basicsSummaryInstallation and Environment SetupC++SubmissionProblem Set 1 — Image class, point operations, and colorSummaryThe Image ClassBrightness and ContrastMore Image Class MethodsColorspacesSpanish Castle IllusionWhite BalanceProblem Set 2 — Convolution and the bilateral filterSummarySmart AccessorBlurringDenoising using Bilateral FilteringExtra creditSubmissionProblem Set 3 — Denoising and demosaickingSummaryDenoising from a sequence of imagesDemosaicingEdge-based greenRed and blue based on green6.865 only (or 5% Extra Credit): Sergey Prokudin-GorskyExtra credit (maximum of 10%)Problem Set 4 — High dynamic rangeSummaryHDR mergingTone mappingExtra credit (10% max)Problem Set 5 — Resampling, warping, and morphingSummaryResamplingWarping and morphingExtra creditProblem Set 6 — Homographies and manual panoramasSummaryClass MorphHomogeneous CoordinatesLinear AlgebraWarp and Image with a HomographyCompute Homography from 4 Pairs of PointsBounding boxesExtra Credit (up to 10% total)Problem Set 7 — Automatic panoramasSummaryPrevious Problem Set CodeClass MorphHarris Corner DetectionDescriptor and correspondencesRANSACAutomatic panorama stitchingBlendingMini planet6.8370: Stitch N Images (6.8371: Extra Credit 5%)Make your own panoramaExtra credits (10% max)Problem Set 8 — Non-photorealistic renderingSummaryPaintbrush splattingPainterly renderingOriented painterly renderingYour imagePaper Review (6.865 only)Extra creditsProblem Set 9 — Make-your-own, video, and ethicsSummaryMake Your Own AssignmentEthical issues in computational photographyAssignment ListsEXIF and image metadataWhat EXIF isThe fields, grouped by what they describeHow far to trust itPrivacy: the metadata that follows the pictureReading and writing EXIFDNG: the Digital NegativeWhat DNG is, and the problem it solvesInside the containerWhat the raw payload looks like: mosaic vs linearThe color recipe: matrices, profiles, and white balanceOpcodes: corrections the decoder must applyCompression, and embedding the originalWhere you meet DNG: adoption and relativesTrade-offs, and DNG's relation to EXIFRendering a raw DNGThe pipeline, step by stepWhere Lightroom's "look" actually comes fromWhat we would ask an Adobe engineer to checkTwo kinds of DNG, and the special casesEasy mistakes (most of which we made)DatasetsClassification and featuresSuper-resolutionDeblurring and restorationDenoisingHDR and tone mappingRetouching and enhancementDepth and motionLight fieldsColor and white balanceFacesInpainting, segmentation, and mattingImage qualityA camera-feature wish listExposure, ISO, and dynamic rangeBracketing more than exposureFocus and depth of fieldComputational raw and the sensorMotion data, metadata, and workflowPanorama and multi-shotThe interface and the ecosystemHow this book was createdTwo documents, not oneCompiling a sectionFigures as codeGenerative imagery: cover art and 3DVerification and reviewKeeping a long book coherentThe toolchainWhat the machine did, and what it did notWho wrote what: a per-part estimateThe course tutor: a local, book-grounded AI teaching assistantWhat it is, and what it is forLocal-firstGrounded in the book: retrieval-augmented generationIt links, it shows equations, it shows figuresTwo front-ends, one coreWhat the instructor seesPrivacy and candorThe semi-automatic grading systemThe shape (to be confirmed)Automatic versus human (to be confirmed)To be filled in (from the instructor)Under the hood: prompts, patterns, and verifiersPrompt patterns that made it workThe verifier suiteWhy this is the interesting partReading a Lytro light fieldThe container: the LFP/LFR formatThe pipeline, step by stepFrom hexagonal lenslets to a uniform gridWhy a white imageOpen questions and where we approximateEasy mistakes (most of which we made)File conversion toolsJPG → PNG, without an alpha channelDNG → a linear PNGA Lytro capture → a Stanford-style light-field archiveHow they run in the browserEasy mistakesThe interactive figures — a prompt-by-prompt making-ofSummaryLorentz resonance — fig-lorentz-resonance (Figure 2.1.8)Rainbow droplet — fig-rainbow-droplet (Figure 2.1.14)Diffraction wave simulation — fig-diffraction-wave-sim (Figure 2.1.29)Exposure-triangle simulator — fig-exposure-triangle-sim (Figure 2.11.5)Repeated quantization — fig-repeated-quantization (Figure 3.1.4)JPEG generation loss — fig-jpeg-generation-loss (Figure 3.17.9)Exposure round-trip (JPEG) — fig-exposure-jpeg-roundtrip (Figure 3.17.10)1-D sampling pipeline — fig-sampling-1d-demo (Figure 3.10.4)2-D sampling pipeline — fig-sampling-2d-pipeline (Figure 3.10.6)Mitchell–Netravali bicubic (B,C) — fig-bicubic-bc (Figure 3.11.13)Rotation resample challenge — fig-rotation-resample-interactive (Figure 3.11.18)Mini-Lightroom — fig-mini-lightroom (Figure 3.18.4)Poisson blending — fig-poisson-blend (Figure 5.1.5)Bilateral grid (3-D) — fig-bilateral-grid-3d (Figure 5.2.15)Perspective montage — fig-perspective-montage (Figure 6.6.13)Beier–Neely morph — fig-beier-neely-demo (Figure 6.3.4)Live face landmarks — fig-face-landmarks-live (Figure 7.10.2)Lens optimizer — fig-lens-optimizer-demo (Figure 9.4.4)Full auto-panorama — fig-pano-stitch (Figure 10.7.9)CLIP-IQA curation — fig-clip-iqa (Figure 11.3.2)Photobio time-lapse — fig-photobio-demo (Figure 11.7.2)Refocus shift geometry — fig-refocus-shift-geometry (Figure 13.4.2)Chroma key — fig-chroma-key (Figure 8.11.7)Portrait-lighting simulator — fig-portrait-lighting-sim (Figure 2.11.26)Aberration explorer — fig-aberration-explorer (Figure 9.1.6)Interactive demo indexSharing, linking, and embedding a demoIntroductionFundamentals — light, optics, sensors, colorBasic image processing and the ISPComputational tools — machine learning and diffusionEdges matter — gradient-domain and edge-preservingWarping and morphingMatching pixels across space and timeSingle-image computational photographyOptics, lenses, and aberration correctionMultiple-exposure imaging — HDR and panoramasMany images and photo collectionsVideoLight fields and plenoptic cameras3-D and depthAppendices and end matterNot yet placedMISSING STUFF, BUGSfig-learning-transition (hand-designed → learned → generative spectrum) — parked hereDenoising by averaging is after basic denoisingPhotomosaics, my self organizing mapsRephotographyExtreme low lightTilt shift,Misc.De-weathering (fog, rain)Near-infrared (NIR) photography (relocated from REVEALING THE INVISIBLE, 2026-06-28)

BACK MATTER

Glossary Acronyms Index

1.0 Intro⧉

In this part

_This part does not have a written introduction yet._ Its chapters and sections:

Contents of this part

1.1 From Digital to Computational Photography
1.2 How to read this book
- 1.2.1 Machine-readable by design
1.3 How this book was made
1.4 What programming language for computational photography?
- 1.4.1 On the desktop: Python, C++, and Halide
- 1.4.2 In the browser: JavaScript and the web stack
- 1.4.3 On the phone: Android and iOS
- 1.4.4 Libraries
- 1.4.5 Vibe coding
1.5 Problem sets

▸ full collapsible outline of this part

1.0 Intro🔗⧉

Contents of this part

1.0 Intro⧉