VIST 375: Human Vision and Digital Image Processing

Cornea

Camera lens (focusing incoming light)

Iris

Shutter or Aperture control (Controls the amount of light)

Pupil

Aperture (the opening that light enters from)

Retina

Film, or image sensor (Receive light)

Optic Nerve

Data cable (transmits visual information to the brain for processing)

Visible Spectrum

Humans can only see a range of wavelengths from red (longer wavelengths) to violet (shorter wavelengths)

Rods

Sensitive to low light, providing scotopic (night) vision. More abundant in the peripheral regions of the retina

Cones

Sensitive to color and detail, providing photopic (daytime) vision. More abundant in the central region of the retina (fovea)

S-cones

Sensitive to short wavelengths, blue wavelengths

M-cones

Sensitive to medium wavelengths, green wavelengths

L-cones

Sensitive to long wavelengths, red wavelengths

How are analog (continuous) signals transferred to digital (discrete) signals?

Sampling, Quantization, and Encoding

Sampling

Measuring the analog signal at regular intervals

Quantization

Converting the sampled values into discrete values

Encoding

Convert the quantized value to binary codes

Quantization Resolution

The number of discrete levels used to approximate the sampled value. (bits or bins)

Spatial Resolution

Pixels sampled per image space (PPI or DPI)

Grayscale Image

An intensity function defined on a rectangular region, or discretely, a 2D matrix

Color Image

Can be considered as 3 grayscale images (R, G, B, channels respectively) stronger channel --> stronger intensity (white)

RGB

Additive color model used for monitors or other digital mediums that emit photons

CMY

Subtractive color model starting with a default (0, 0, 0) white medium

CMYK

Color model mostly used for printing, adds black in K

HIS, HSL, HSV

Hue [0, 360], saturation, L/V/S brightness. Better matches human color interpretation

Lossless Compression

Offers perfect reconstruction. PNG, TIFF, BMP, GIF

Lossy Compression

Sacrifices some quality for greater size reduction. JPEG, MPEG

Dynamic Range

Difference between its max and min pixel values

Variance

A pixel's average distance from mean intensity, standard deviation

Image Histogram

Visualizes an image's intensity distribution by plotting the number of pixels for each pixel value

Histogram equalization

multiple histograms can be applied to a single image allowing the viewer to focus on areas of differing subject contrast

Correlation

Compute the sum of dot products in a pixel wise manner for each sliding window, producing a new pixel value at each position

Convolution

Similar to correlation, only flipped

Low-pass Filters

Allow low-frequency info to pass, filter out high frequency information. Keeps global info.

Low-pass Filters examples

Box, averaging, gaussian

High-pass Filter

Allow high-frequency info to pass, filter out low-frequency information

High-pass filter examples

Derivative, Laplacian, edge detectors, sharpening

Padding

Adding fake cells around input image in order for the mask to work on all pixels of the image

Edge Detection

Discontinuity of intensities in the image

Upsampling

Increase spatial resolution; usually through interpolation (nearest-neighbor)

Downsampling

Decrease spatial resolution; in order to avoid aliasing, smooth before downsampling

Deep Neural Networks

AI, ML, DL, and Neural Networks

Multi-Layer Perceptron (MLP)

Neuron (node) with computing capabilities: linear combination, nonlinear activation. Built by stacking multiple layers with multiple nodes in each layer

Convolutional Neural Networks (CNNs)

A type of neural network that uses convolutional layers to learn features from images

Piecewise curves

consecutive line segments calculated with linear interpolation from one point to the next

Primary rendering techniques in CG

Rasterization, Ray tracing: two classical techniques
Neural rendering: more recent addition

Rasterization

Forward approach that directly projects 3D points to 2D viewport.

3D to 2D rasterization pipeline

1. Viewing transformation (place camera in 3D space)
2. Model transformation (move objects to position within view volume)
3. Projection transformation (project 3D points onto 2D image plane)
4. Viewport transformation: Map image plane to actual coordinates on the screen's viewport

Face Normal

Perpendicular vector to a triangle's surface.

Vertex Normal

Weighted average of adjacent face normals.

Flat Shading

Single color per face, no interpolation.

Gouraud Shading

Interpolates colors at face vertices. Smoother appearance

Phong Shading

Interpolates normals, computes pixel-by-pixel color. Smoothest and most accurate

Ambient Reflection

Global illumination from all light sources.

Diffuse Reflection

Scattered light from rough surfaces. Depends on light position but NOT viewer position

Specular Reflection

Concentrated light reflection from shiny surfaces. Depends on light position AND viewer position

Ray Tracing

Backwards approach simulating light paths. Shoot rays from the eye through each image pixels. Casts secondary rays to simulate extra reflection/refraction

Ray Casting

Shooting rays through image pixels. Hard shadows, less accurate reflections and shadows

Neural Rendering

Data-driven rendering using neural networks.

Neural Implementation of Classic Rendering Pipeline

Modeling: geometry no longer needs meshes; but is implicitly encoded in neural networks
Texturing: No longer uses bitmaps and texture mapping, now directly generated
Lighting: No longer uses shading models; now typically learned together with geometry from the data

NeRF Representation

5D function encoding 3D scenes in a radiance field

Radiance Field Dimensions

3 dimensions for spatial position (x, y, z)
2 dimensions for view direction (theta, trident)

Volume Intensity σ

Depends on 3D position (x,y,z)

Color c (r,g,b)

Depends on both 3D position (x,y,z) and view direction

Training a NeRF

Compare original photo I with rendered photo I'

Differentiable Rendering

Refines 3D representation via image comparison.

Gaussian Splatting

Enhanced point cloud representation with Gaussians.

Gaussian parameters

Position, Covariance, Color, and Alpha

Training 3DGS

Structure-from-Motion (SfM) that Estimates camera trajectory and 3D points using videos

Autoencoder (AE)

Deterministic model for unsupervised learning. Lacks latent structure

Variational Autoencoder (VAE)

Probabilistic model mapping to distributions. Has more structure but more complex and poor generation quality

Latent Space

Representation of compressed data features.

Generative Adversarial Networks (GANs)

Two networks generating and classifying data. G plays a game against D to fool it into believing its fake data is real. High quality but more complex and hard to train

Diffusion Models (DMs)

Simulates diffusion to create new data. Two processes of adding noise to data, and a reverse process that learns to remove the noise and reconstruct the original data or make new data