Computer Vision L1-L4

Computer Vision Definition

Automatic understanding of images and video

1. Measurement

2. Perception and Interpretation

3. Search and Organization

4. Visual Generation

Vision for Measurement

• computing properties of the 3D world from visual data

• Examples: real-time stereo, structure from motion, tracking

Vision for Perception and Interpretation

• algorithms and representations to allow a machine to recognize objects, people, scenes, and activities.

• Example: recognizing objects in a photo

Visual Search and Organization

• algorithms to mine, search, and interact with visual data

• Example: social media algorithms

Visual Generation

• algorithms for manipulation or creation of image or video content

• Examples: photo filters, ai generated images/video

Computer Vision Uses

• Optical character recognition (OCR)

• Face Detection

• Face Recognition

• Biometrics (fingerprint scanner, Face ID, iris patterns)

• Visual Effects (CGI, de-aging)

• Sports Overlays for Live

• Robotics

• Smart and Self Driving Cars

• Games (Kinect)

• Medical Imaging

Why is Computer Vision difficult

• Ill-posed problem: real world much more complex than what we can measure in images (3D to 2D)

• Impossible to “invert” image formation process

Challenges in Computer Vision

• viewpoint variation

• scale

• illumination

• intra-class variation (same object that looks different; cars)

• motion

• background clutter

• occlusion (object partially hidden

• local ambiguity (blurry images that have similar looking shapes but are different objects)

How far are we?

• Current models are very good at the “easy” things

• Humans are still better at the “hard” things

• Deep learning keeps moving the line between the two

Geometric Primitives in 2D and 3D

• points, lines, and planes

2D Lines

• L = ax + by + c

2D Linear Transformations

• only 2D linear transformations can be represented by a 2×2 matrix

• Combinations of

  • scaling

  • rotation

  • shear

  • reflection/mirror

Scaling

• multiplying each of its components by a scalar

  • uniform means scalar is same for all components

  • non-uniform means different scalars per component

• 2D Equation

  • x’ = ax

  • y’ = by

Rotation

• 2D Equation

  • x’ = x cos θ - y sin θ

  • y’ = x sin θ + y cos θ

Shear

• slant or skew shape by one side parallel

• 2D Equation

  • x’ = x + (shx * y)

  • y’ = (x * shy) + y

Reflection/Mirror

• 2D Equation

  • For X-axis: y’ = -y

  • For Y-axis: x’ = -x

  • For origin: x’ = -x, y’ = -y

Translation

• 2D Equation

  • x’ = x + tx

  • y’ = y + ty

Homogeneous Coordinates

Matrix Composition - what, why, note

• transformations can by combined by matrix multiplication

• efficiency is main point of composition

• multiplication order does matter

2D Affine Transformations

• combinations of linear transformations and translations

• note: parallel lines remain parallel

2D Projective Transformations

• combinations of affine transformations and projective warps

• only lines remain lines

Classification of 2D Transformations

• each variable in the matrix accounts for one degree of freedom

Classification of 3D Transformations

Focal Length

distance from center of camera to image plance

Bare - Sensor Images

results in blurry image because all light rays are collected for image plane

Pinhole camera

• blocks most rays so image can form

• flipped image plane as side effect

Vanishing Point

• in images, parallel lines on the same plane lead to collinear vanishing points (horizon)

• was a great way to identify ai generated images, but is being fixed in recently

Perspective Projection

• size of object in image plane is inversely proportional to distance

  • closer the object to camera center, it appears larger in the image

  • far objects appear smaller than close ones

• the larger the focal length, the bigger the image

• From 3D space to 2D point

Perspective Projection Math

• f is focal length and z is length of object to camera center

• Points Defined in camera coordinate system

• P = [ x y z ] projects to 2D image point P’ = [ x’ y ] where

  • x’ = f’ ( x/z )

  • y’ - f’ ( y/z )

• P = [ x y z 1 ] projects to 2D image point [ x’ y’ 1 ] where P’ = CP

• 
  

Perspective Projection Calibration Matrixes

• sensor and pinhole aligned

• shifted sensor

• non square pixels and skewed sensor
  
  

Projective Camera Complete Matrix

• calibration, projection, extrinsics

• 11 degrees of freedom

What does projection lose?

• depth

• length

• angles

What does projection preserve?

• straight lines

Weak Perspective

• x’ = m x

• y’ = m y

• m = f’ / z0

Orthographic Projection

• 1 to 1 projection of 3D object to 2D

• x’ = x

• y’ = y

Types of Projection

• perspective

• weak perspective

• orthographic

• application and situation determine use case

Three Parts of Photometric image formation

• lighting

• reflectance and shading

• optics

Lighting

• 2 kinds; point or area

• shadows can help determine if image is ai generated

Reflectance and shading

• surface reflection depends on both the viewing and illumination direction with Bidirectional Reflection Distribution Function

• also a good way to tell if image is ai generated

Specular Reflection

incident light rays are reflected in a direction that is rotated by 180 around the surface normal

Diffuse Reflection

• Light scatters in all directions

  • Lambertian

  • Oren-Nayar

  • Phong

  • Fully general: BRDF or Bidirectional Reflection Distribution Function

Optics

light source reflects off of object and hits camera sensors to determine color and bightness

Pinhole Aperture/ Size

• large lets too much light in so blurry

• small lets to little light in so fuzzy

Lens

• lens focuses light onto the film

• rays passing through the center are not deviated

• all parallel lines converge to one point on a plane located at the focal length f

• if half pinhole covered it is a blurry image because not enough light went through. with a lens, all light goes through so focused full image

Thin lens model

• zi = distance from image plane to lens, adjustable

• zo = distance from lens to object/light source

• 1/zo + 1/zi = 1/f

• must ensure that zi does not equal f, focus at infinity

• when zi > f, brings back focal plane from infinity

Digital Camera

• replaces film with sensor array

• each cell in array is a light sensitive diode that converts photons to electrons

• energy efficient and good quality

• two types:

  • charge coupled device (CCD)

  • CMOs

Sensing Pipeline

Sampling to prevent Aliasing

• sampling frequency needs to be twice as high as highest signal or there will me artifacts

• fs >= 2 fmax

Bayer Grid

• sensors are color blind, we add filters to detect color. one filter per pixel

• effects

  • could lose some color information

  • sampling at lower resolution

  • put higher resolution in green because human perception is more sensitive to it

Images in Python

• represented as matrix

• NxM RGM image called im

  • im(0,0,0) = top left pixel value in R channel

  • im (y, x, b) = y pixels down, x pixels to the right in bth channel

  • im(N-1, M-1, 2) = bottom right pixel in B channel

Color Spaces

• RGB

• HSV

• YCbCr

• L*a*b

• most information in intensity of light not color

RGB

• red, green, blue

• default color space

• drawbacks:

  • strongly correlated channels

  • non perceptual

HSV

• Hue, Saturation, Value

• intuitive color space

YCbCr

• fast to compute

• good for compression

• used by TV

L*a*b

perceptually uniform color space

Types of Image Transformations

• warping - changes domain of image function, changes location

• filtering - changes range of image function, changes color

Types of Filtering

• point operation - single pixel input

• neighborhood operation - multiple pixel input

Point Processing Examples

• Image result = I’(X,Y)

• contrast = a I(X,Y), a

• brightness I(X,Y) + b,b

• histogram equalization = h(I(X,Y))

• arithmetic (example image 1 - image 2)

• composting

  • F = image

  • a = alpha map

  • aF = foreground

  • aF + (1 - a)B = changing background

Human Vision

• eye is pinhole camera

• iris - colored annulus with radial muscles

• pupil - hole whose size is controlled by iris

• retina - back of eye with contains the photoreceptor cells (rods and cones)

• cones - less sensitive, operate in high light, color vision

• rods - highly sensitive, operate in low light, gray-scale vision