Mobile Visual Computing (CSC 292/572) Fall 2022

Tentative Schedule

8/31	Class Canceled (Instructor Travel)
9/2	Introduction and Overview Suggested Reading: The Rise of Mobile Visual Computing Systems (IEEE Pervasive Computing, 2016)

Preliminaries

9/7	Geometric Transformations Buzzwords: Affine transformation, transformation matrix, scaling, rotation, identity matrix, translation, camera projection, matrix decomposition, singular value decomposition, unitary matrix, orthogonal vector Required Reading: Chapter 6, Fundamentals of Computer Graphics (4e). We didn't discuss materials in 6.1.2, 6.1.4, 6.2.1, 6.2.2, 6.4; skip or skim them if you want. Chapter 5, Read Fundamentals of Computer Graphics (4e) if you need to refresh basic linear algebra. Study 5.4 carefully if you haven't seen that material before.
9/9	Fourier Analysis Buzzwords: Inner product, projection, Fourier Series, Fourier Transform, Discrete Fourier Transform, spectrum, low-pass filter, high-pass filter Required Reading: Chapter 2.1, 2.2, Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Focus on getting the idea; skip the math proofs if you want; pay attention to how they see Fourier Transform from a vector-space perspective. Skim 2.6 if you want; we will come back to this later.
9/14	Sampling and Convolution Buzzwords: Nyquist-Shannon sampling theorem, signal sampling, signal reconstruction, convolution, convolution theorem, compressive sensing Required Reading: Chapter 9.1, 9.2, and 9.6, Fundamentals of Computer Graphics (4e). Skim other chapters in 9. The FFT code I used in the class to filter images. Run it and understand how it works. Suggested Reading: Image Filtering Tutorial. Nice illustrations of filtering/convolutions. Has more filter types than what we discussed in the class.

Human Visual System

9/16 9/21

Spatial and Color Vision

Buzzwords: Eyes, retina, fovea, foveated rendering, photoreceptors, cones, rods, blind spot, light spectrum, spectral power distribution, photopic vision, scotopic vision, luminous efficiency function, lumen, cone fundamentals, color blindness, dichromat, unilateral dichromat, isochrome, tetrachromacy, invisible light, visual cortex, metamerism, metamer, color matching experiment, color matching function, trichromatic theory of color, color space

Required Reading:

Chapter 1 up until (and include) 1.4.1, Principles of Digital Image Synthesis, which talks about human visual system are great. Read the rest of Chapter 1 if you like. We won't cover them except Chapter 1.7, which we will do next lecture.
Interactive tutorial exploring the connection between color matching functions and cone fundamentals.
Chapter 19 up till (and include) 19.1.4, Fundamentals of Computer Graphics (4e); skip the XYZ space discussion in 19.1.4, which we will discuss next time.

Suggested Reading:

Interactive tutorial exploring color blindness.
How the CIE 1931 RGB Color Matching Functions Were Developed from the Initial Color Matching Experiments. An article I wrote that explains how exactly the color matching functions are derived from the color matching experiments.
Chapter 2A, Billmeyer and Saltzman's Principles of Color Technology.

9/23

Colorimetry

Buzzwords: Colorimetry, color cube, gamut, quantization, chromaticity, chromaticity plot, chromaticity gamut, XYZ color space, reference white, black-body radiation, Planckian locus, color temperature, correlated color temperature, sRGB, additive color model, subtractive color model, perceptual color space, HSV, HSL, HSB, hue, saturation, value, lightness, brightness, CMYK, gamma, MacAdam ellipses

Required Reading:

Finish the rest of Chapter 19.1 plus 20.2.2, Fundamentals of Computer Graphics (4e).
Chromaticity, Gamut, and the Scary World of Imaginary Colors.
Building a Color Cube.

Suggested Reading:

Chapter 4A-4D, Billmeyer and Saltzman's Principles of Color Technology. Study carefully Figure 4.23, which is great. Skip materials that I didn't cover in the lecture if you want.
Chapter 19.2, Fundamentals of Computer Graphics (4e). In the lecture we didn't talk about CIELAB in Chapter 19.2.4 in detail, but skim it if you want.

Color in Nature, Arts, and Technologies

9/28 9/30

Color Sources & Color From Emission: Bioluminescence, Display, and Lighting

Buzzwords: Thermal radiation, bioluminescence, electroluminescence, biofluorescence, absorption, scattering, emission, transparent, translucent, opaque, reflectance, glazing, vanish, incomplete hiding, spectroradiometer, spectrophotometer, fluorescent light, phosphor, color rendering index, emissive display, light-filtering display, LED, LCD, liquid crystal, polarizer, white LED, phosphor-converted LED, field sequential color, FSC, color wheel, DLP projector, digital micromirror device, DMD, color breakup, three-chip projector, motion blur, solid angle, irradiance, radiance, nits, lux, brightness, dynamic range

Required Reading:

Suggested Reading:

Human Factors in Lighting. Chapter 1.6.3.2 talks about Color Rendering Index. Chapter 1.7.3 talks about at a high-level how different light sources work. Chapter 1.7.4 compares across light sources using a variety of metrics. Pay attention to Table 1.4.
Watch a video of CBS's FSC TV system at work.
Measured color space of displays by ClarkVision.

10/5 10/7

Color From Absorption & Scattering: Painting, Dyeing, and Printing

Buzzwords: absorption coefficient, absorption efficiency, cross section area, scattering coefficient, scattering efficiency, pigment, binder, refractive index, exponential decay, exponential growth, phase function, multiple scattering, Kubelka-Munk model, sub-surface scattering, glossy, specular, diffuse, mass tone, tint, saturation

Required Reading:

Chapter 5.7, Digital Modeling of Material Appearance. It has a nice discussion radiometry metrics and emission/absorption/scattering. Physically-based as far as rendering is concerned.

Suggested Reading:

Chapter 18.1 (except 18.1.6) and 18.3, Fundamentals of Computer Graphics (4e). Radiometry and photometry; just another narrative if you want.

Digital Camera Imaging

10/12	Camera Optics Buzzwords: Pinhole camera, motion blur, convex lens, concave lens, Snell's law, refraction index, gravitational lens, waveguide, thin lens approximation, paraxial approximation, gaussian lens equation, circle of confusion, depth of field, F-number, magnification factor, field-of-view, think lens, optical aberration, chromatic aberration, monochromatic aberration, spherical aberration, motion from blur, depth from blur Required Reading: Chapter 4.2 -- 4.5, Virtual Reality. Talks about cameras, lenses, and optical aberrations at a high level; a pleasant read. Suggested Reading: Archimedes and the Burning Mirror. Chapter 5, Optics (5e). Geometrical optics very well explained. Even studying the illustrations alone can get you pretty far.
10/14	Image Sensor Basics Buzzwords: IR filter, UV filter, microlens, photoelectric effect, photodiodes, quantum efficiency, light-matter interaction, energy-harvesting camera, pixel well, full-well capacity, saturation, dynamic range, high dynamic range, NIR photography, thermographic camera, IR camera, depth sensing, fill factor, charge-to-voltage conversion, voltage gain, CCD sensor, CMOS sensor, rolling shutter, global shutter, analog-to-digital conversion, ADC, resolution Required Reading: Chapter 2 of Image Processing for Embedded Devices From CFA Data to Image Video Coding. A good overview of camera optics and sensor in general. No math. Suggested Reading: Digital Camera System on a Chip. Read an overview of modern CMOS camera sensors from the person who invented it. A slightly more technical version is here. Evolving Image Sensor Architecture through Stacking Devices. Good overview of recent advancements of 3D-stacked image sensors. Chapter 3 of Image Sensors and Signal Processing for Digital Still Cameras. It has an in-depth discussion of the various aspects of camera sensors. What's relevant for today's lecture is Chapter 3.1 and 3.2. You can safely skip the math there, but going through it gives you a nice technical understanding of the topic.
10/19	Noise and Color Sensing Buzzwords: noise, temporal noise, fixed pattern noise, photon shot noise, Poisson distribution, dark current noise, read noise, reset noise, hot pixels, bias/offset voltage, noise amplification, signal to noise ratio, signal gain, ISO, ADC gain, ADC noise, ADC quantization error, correlated double sampling, recover noise-free images, dark-frame subtraction, bias frame, dark frame, flat-field correction, dynamic range, stop, dB, three chip sensor, foveon sensor, bayer pattern, spectral sensitivity function, SSF Required Reading: Noise, Dynamic Range and Bit Depth in Digital SLRs. Read just that first page; an information treatment of different sources of noise. Suggested Reading: ARVR Sensing. Has good slides on image sensor hardware; includes some discussions on Digital Pixel Sensor (DPS), Correlated Double Sampling (CDS), and Time-to-Saturation for high dynamic range imaging. Noise and modelling SNR. A great technical discussion on noise. CCD Noise Sources and Signal-to-Noise Ratio. Another narrative of sensor noise. Talks about CCD, but the general idea is the same.
10/21	Camera Image Signal Processing Buzzwords: image signal processing, ISP, denoising, filtering, Gaussian filter, low-pass filter, bilateral filter, demosaic, color correction, regression, ColorChecker Required Reading: A Software Platform for Manipulating the Camera Imaging Pipeline (ECCV 2016). It's an open-source ISP pipeline implementation. Our discussion doesn't exactly follow and we will leave a few stages (e.g., tone mapping, white balancing) to later lectures. Exploring camera color correction. An interactive tutorial that we developed that walks you through camera color correction. Suggested Reading: Andrew Rowlands' Color conversion matrices in digital cameras: a tutorial is the definitive to camera color correction; give it a read if you are serious about it, which also talks about the connection between color correction and white balance. These three articles in a series (1, 2, 3) has a good explanation of color correction too.
10/26	Introduction to HDR Imaging and Tone Mapping Buzzwords: HDR imaging, tone mapping, graduated neutral density filter, GND, multiple exposures, radiance map, linear tone mapping, global tone mapping, local tone mapping, contrast, local contrast, HDR look Required Reading: Tones and Contrast and Tone Curve in Photoshop. They talk about the notion of tone and contrast as well as how changing image tones on a standard DR image changes the visual effects. An interesting thing to note is that tone mapping in SDR images usually uses an S-shape curve that increases the contrast in midtones and compress contrasts in highlights and shadows, whereas tone mapping curves for HDR images tend to be inverse-S shape to bring out details in highlights and shadows. Suggested Reading: Burst photography for high dynamic range and low-light imaging on mobile cameras (SIGGRAPH Asia 2016). The HDR+ pipeline in Google Pixel and Pixel 2. Spatially Nonuniform Scaling Functions for High Contrast Images (SIGGRAPH Asia 2016). Early work on exploiting local contrast in tone mapping. Recovering High Dynamic Range Radiance Maps from Photographs (SIGGRAPH 1997).
10/28	Feedback Loops in Cameras: The 3A Algorithms Buzzwords: Feedback-driven optimization, color constancy, chromatic adaptation, neutral point, white point, illuminant, von Kries adaptation model, auto white balancing, AWB, active AF, passive AF, phase detection, contrast detection, PDAF, CDAF, cross-correlation, autofocus point, dual pixel AF, time-of-flight, depth from stereo, structured light, metering Required Reading: Play with the Canon exposure control simulator. Chapter 19.3 and 19.4, Fundamentals of Computer Graphics (4e). It provides a gentle introduction to chromatic adaptation and color appearance model. Suggested Reading: Principles and Practices of Chromatic Adaptation. The theory behind basic chromatic adaptation models and their applications in color imaging. Synthetic Depth-of-Field with a Single-Camera Mobile Phone (SIGGRAPH Asia 2018).
11/2	Astronomical Image Calibration (by Kelly Douglass)
11/4	Image and Video Compression Buzzwords: Encoder, decoder, codec, lossless, lossy, luma, chroma, chroma subsampling, Y'CbCr, JPEG, DCT, basis function, quantization matrix, entropy encoding, run-length encoding, RLE, Motion-JPEG, M-JPEG, MPEG, H.262, H.264, H.265, AVC, HEVC, container former, .mp4, intra-frame prediction, inter-frame prediction, motion estimation, motion compensated interpolation, three step search, residual, macroblock, motion vector, reference frame, I-frame, P-frame, B-frame, GoP, sub-pixel motion, transcoding Required Reading: Chapter 16, Digital Video and HD : Algorithms and Interfaces. An high-level introduction to video compression, not tied to a particular standard. Read the example on page 496-499 in Chapter 45, Digital Video and HD : Algorithms and Interfaces. A JPEG compression walk-through using a different example from what I used in the lecture. Suggested Reading: Chapter 3, The H.264 Advanced Video Compression Standard. A different, but very good, narrative of (H.264) video encoding. vbench: Benchmarking Video Transcoding in the Cloud (ASPLOS 2018). Profiling the video transcoding system in Google. Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision (ISCA 2018). How motion (naturally generated by cameras) can be used to help computer vision tasks.
11/9 11/11	Immersive Visual Content: Panorama, (Stereo) 360° Videos, and Light Field Buzzwords: Panorama, field of view, FOV, telephoto lens, perspective projection, ghosting, cylindrical projection, spherical projection, distortion, equi-rectangular projection, cubemap projection, spherical projection, stereo 360, omni-direction stereo, light field, extended depth of field, plenoptic camera, NeRF, photogrammetry, structure from motion, bundle adjustment, ego-motion estimation Suggested Reading: A high-level overview of Facebook Surround 360 algorithm pipeline. Bringing pixels front and center in VR video. This Google post talks about the issues with equirectangular projection and introduces the EAC projection. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Use neural networks to estimate light field.

Modeling and Rendering

11/16	Graphics Overview and Introduction to 3D Modeling Buzzwords: Photo-realism, computer-generated imagery, parametric surface, parametric curve, polygon mesh, triangle mesh, point cloud, point set, normal, polygon triangulation, bezier curve, bezier surface, basis functions, Bernstein polynomials, LiDAR, point set triangulation, Delaunay triangulation Required Reading: Read the Scratchapixel articles on Bezier curve and Bezier surface. Feel free to ignore materials that we didn't discuss in the lecture (e.g., joining Bezier curves). Chapter 12.1 up till 12.1.1, Fundamentals of Computer Graphics (4e). Suggested Reading: The rest of Chapter 12.1, Fundamentals of Computer Graphics (4e). Read the Scratchapixel articles on Introduction to Polygon Meshes.
11/18	Rasterization and GPU Buzzwords: rasterization, ray tracing, image-centric, object-centric, visibility, shading, local frame, world frame, camera space, raster space, perspective projection, orthographic projection, environmental camera, near-clipping plane, far-clipping plane, barycentric coordinates, depth-buffer, z-buffer, supersampling, GPU, shader, SIMD Required Reading: Read the first four chapters on this Scratchapixel lession on Rasterization: a Practical Implementation. Both this Scratchapixel article and Chapter 7.2 and 7.3 of Fundamentals of Computer Graphics (4e) talk about how to calculate the perspective transformation matrix. They use different conventions but the principle is the same. Go over them to reinforce your understanding. Pay attention to details. Suggested Reading: Chapter 8.6Principles of Digital Image Synthesis. A good discussion of supersampling. Chapter 2.7, Fundamentals of Computer Graphics (4e). It provides a pretty detailed discussion of barycentric coordinates for triangles. Chapter 4.2, Fundamentals of Computer Graphics (4e). Gives some intuitions behind different projections. GPU Computing (Proceedings of the IEEE, 2008). Not the most recent article, but still provides a good overview of the transition from GPU to GPGPU.
11/23	Thanksgiving Holiday
11/25	Thanksgiving Holiday
11/30	Ray Tracing and Acceleration Structures Buzzwords: ray-triangle intersection, acceleration structures, space partition, object partition, uniform grid, octree, k-d tree, binary space partitining tree, BSP, bounding volume hierarchy, BVH, recursive ray tracing, whitted-style ray tracing Required Reading: An Overview of the Ray-Tracing Rendering Technique. A brief overview of Whitted's (recursive) ray tracing algorithm. We haven't talked about shading yet, but skim through it; you will get a nice idea of it. A Ray-Tracing Pioneer Explains How He Stumbled into Global Illumination. Turner Whitted's retrospective views of how the recursive idea came along and how it was positioned from today's perspective. Suggested Reading: Introduction to Ray Tracing: a Simple Method for Creating 3D Images. Chapter 4.3, Physically Based Rendering. A detailed implementation of BVH.
12/2	Shading Buzzwords: Phong model, global illumination, ambient shading, rendering equation, bidirectional reflectance distribution function, BRDF, Monte Carlo integration, Monte Carlo estimator, variance, bias, specular lobe, caustic, bidirectional subsurface scattering function, BSSDF, texture mapping, texel, UV unwrapping, UV map, UV island, texture sampling, texture filtering Required Reading: Monte-Carlo integration (from Scratchapixel). Introduction to global illumination and path tracing (from Scratchapixel; with code). Suggested Reading: Chapter 11.4 of Fundamentals of Computer Graphics (4e) gives an overview of the applications of texture mapping. Read other sections in the same chapter if you care about the math behind texture mapping. Chapter 14.3 of Fundamentals of Computer Graphics (4e) gives a slightly formal treatment of Monte-Carlo integration. Chapter 14.1 and 14.2 review some basic concepts. Chapter 14.4 concerns with random sampling (which is tricker than you think). Deriving Lambertian BRDF from first principles. After this lecture you should be able to understand the BRDF derivation in this article.

Applications

12/7	Medical Imaging (Guest Lecture by Xiao Wang) Imaging and Rendering in Cultural Heritage Buzzwords: Herculaneum Papyri, Archimedes Palimpsest, Hipparchus’s Star Catalogue, Scheimpflug imaging, pentimento, The Old Guitarist, The Martellus Map, The Hunt-Lenox Globe, multispectral imaging, structure from motion Suggested Reading: Talk on The Archimedes Palimpsest Talk on Reading the Herculaneum Papyri: Yesterday, Today, and Tomorrow
12/9	Virtual Reality and Augmented Reality Buzzwords: Eye piece, eye relief, virtual display, motion to photon latency, motion tracking, DoF, IMU, gyroscope, accelerometer, inside-out/outside-in tracking, teleportation, stereopsis, vergence-accommodation conflict, light field display, gaze tracking, foveated rendering, ventral metarism, foveated ray tracing, lens distortion, optical see-through, video see-through, waveguide Required Reading: Head-mounted Displays and Lenses. Accommodation and Vergence in Head-mounted Displays. Suggest Reading: Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems (Advances in Optics and Photonics, 2018).
12/14	Research in Horizon Lab