Exhaustive Computer Vision and Image Processing Study Guide

Course Administration, Evaluation, and Environment Setup

• Course Identifiers: CV1620 / DPS920 (Computer Vision).

• Instructional Schedule & Mode:

  • Delivery Mode: Hybrid sessions (in-person, online, and recorded).
  • Weekly Contact: $3.5\,\text{hours}$ per week.
  • Tuesday Session: 1:30 PM to 3:15 PM (in-person).
  • Thursday Session: 3:20 PM to 5:05 PM (online).
  • Term Pattern: Typical Summer term delivery.

• Grading Scheme:

  • Assignments: $15\%$
  • Labs: $15\%$
  • Quizzes: $15\%$
  • Midterm Exam: $20\%$
  • Final Exam: $22.5\%$
  • Final Project: $12.5\%$

• Credit Criteria:

  • Achieve $\ge 50\%$ of the overall course mark.
  • Achieve $\ge 50\%$ average across all examinations.
  • Successfully complete the final project.

• Reference Literature:

  • Kenneth Dawson-Howe, A Practical Introduction to Computer Vision with OpenCV, Wiley.
  • François Chollet, Deep Learning with Python, Second Edition, Manning.
  • Adrian Kaehler & Gary Bradski, Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library, O'Reilly.

• Development Environment Setup Procedure:

  1. Install Visual Studio Code (VSCode).
  2. Install Python, executable setup, and ensure Python is added to the system PATH variable.
  3. Add the Python extension in VSCode.
  4. Install virtual environment utility via pip: bash pip install virtualenv      
  5. Verify virtualenv installation: bash pip list # or virtualenv --version      
  6. Create isolated environment using target Python binary path: bash virtualenv -p <path_to_python_version> <env_name> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  7. Activate virtual environment: bash <path_to_project>/<env_name>/Scripts/activate &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  8. Install required scientific compute and vision packages: bash pip install numpy pip install opencv-python pip install matplotlib pip install ipykernel      
  9. Add Jupyter Notebook extension in VSCode.
  10. Register virtual environment kernel to Jupyter: bash python -m ipykernel install --user --name=<venv_name> --display-name "Python <venv_name>" &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
  11. Configure Git repository controls.

Fundamentals of Image Processing and Computer Vision

• Core Definitions:

  • Image Processing: A subfield of signal processing that uses mathematical operations to transform and manipulate digital images. Its primary objective is to enhance image quality (e.g., noise reduction, contrast adjustment) or extract fundamental features (e.g., edge detection, object segmentation).
  • Computer Vision: An interdisciplinary field combining AI, machine learning, and image processing to interpret, analyze, and understand the contents of digital visual media (e.g., object detection, facial recognition, semantic segmentation).

• Historical Evolution:

  • 1960s: Birth of digital image processing. Early developments emerged within electrical engineering and computer science, driven primarily by space exploration. NASA utilized digital image processing techniques to correct and enhance lunar images captured during the Apollo missions.
  • 1970s: Rapid advancements in medical imaging modalities (e.g., CT scans, X-rays) and satellite remote sensing.
  • 1980s: Rise of personal computing and expansion of desktop visual processing.
  • 1990s: Digital multimedia revolution, image compression standardization, and consumer digital cameras.
  • 2000s: Integration of computer vision tools into consumer software and web platforms.
  • 2010s–Present: Modern era defined by the convergence of classical image processing with artificial intelligence and deep learning models (such as Convolutional Neural Networks, post-2012).

• OpenCV Background: Developed originally at Intel to accelerate computational benchmarks on multi-core CPUs. The library is written natively in C++ for maximum computational performance, with bindings provided for Python and Java. Key architect: Gary Bradski.

Human Vision vs. Digital Vision Systems

• Anatomy of the Human Eye:
  • Light passes through the Cornea, Aqueous humour, Pupil (encircled by the Iris), and Crystalline lens (adjusted by Ciliary muscles).
  • Focuses through the Vitreous humour onto the Retina, which converts light into nerve impulses routed to the brain via the Optic nerve.

• System Comparison Metrics:

  • Receptor Elements: The human retina contains approximately $127 \times 10^6$ photoreceptor elements (rods and cones). A digital camera sensor requires a comparable number of discrete sensor pixels to match human visual resolution.
  • Channel Depth: Human vision encodes color via three cone types (sensitive to red, green, blue spectrums). Digital color sensors represent every spatial coordinate using three color values.
  • Temporal Frame Rate: Human vision processes continuous light into a perceived visual stream equivalent to $50\,\text{fps}$ to $60\,\text{fps}$.
  • Binocularity: Dual physical receptors (two eyes / two stereo cameras) provide simultaneous stereoscopic data streams for depth perception.

• Capture Mechanisms: Transitioned from chemical film cameras (relying on atomic/chemical reactions of silver halide emulsions inside light-tight film cartridges exposed in darkrooms) to solid-state digital sensors (CCD and CMOS microchips capturing light-generated electrical charges).

Digital Image Representation and Color Models

• Signal Foundations:

  • Analog Signals: Continuous physical signals containing infinite possible values within a defined range.
  • Digital Signals: Discrete physical signals quantized into finite numeric representations (binary $0$s and $1$s).
  • Visible Light Spectrum: Humans perceive light at wavelengths between $400\,\text{nm}$ and $700\,\text{nm}$. Camera digital sensors are designed specifically to operate within this range.

• Pixel Structure and Coordinates:

  • Pixel (Picture Element): The smallest discrete, addressable component of a digital image storing color or intensity values.
  • Resolution: Total count of pixels in an image matrix, defined as width ($w$) multiplied by height ($h$).
  • Spatial Coordinate System: Structured as a 2D grid matrix where index $(0, 0)$ represents the top-left corner pixel. Horizontal indices represent columns ($x$ axis), and vertical indices represent rows ($y$ axis).

• Color Models and Spaces:

  • Binary: 1 bit per pixel. Pixels are strictly restricted to binary values: $0$ (black) or $1$ / $255$ (white).
  • Grayscale: Single-channel intensity representation. Standard 8-bit unsigned integer (uint8) stores scalar values ranging from $0$ (black) to $255$ (white).
  • RGB Color Model: Additive model combining Red, Green, and Blue channels. Standard 24-bit representation allocates 8 bits per channel ($0$ to $255$ per channel). Total possible color states: $256 \times 256 \times 256 = 2^{24} = 16,777,216$ unique colors.
  • BGR Color Model: Standard spatial channel order used by OpenCV internally, ordering channels as Blue (Index 0), Green (Index 1), and Red (Index 2). (Libraries like TensorFlow and Matplotlib utilize default RGB order).
  • CMYK Color Model: Subtractive color model combining Cyan, Magenta, Yellow, and Key (Black), used primarily in physical color printing processes.
  • HSV Color Model: Cylindrical color representation comprising Hue (color angle/type), Saturation (color purity/intensity), and Value (color brightness).

• File Storage Formats:
  • JPEG (.jpg): Lossy compression algorithm. Reduces file size significantly at the cost of high-frequency visual fidelity. Does not support alpha channel transparency.
  • PNG (.png): Lossless compression algorithm. Preserves exact pixel data and supports full alpha channel transparency.
  • TIFF (.tif): High-fidelity format supporting uncompressed, lossless, or lossy modes with multiple layer storage. Used in commercial publishing and medical applications.

Core Image Operations in OpenCV and NumPy

• Basic I/O Operations:

  • Reading Images:

  import cv2
  import numpy as np
  
  # Reads image into a BGR uint8 NumPy array
  image_array = cv2.imread("Lucy.jpg")
  print(f"Image shape: {image_array.shape}")  # Returns (height, width, channels)
      ```
  - **Displaying Images**:
  

  python cv2.imshow("Window Title", image_array) cv2.waitKey(0) # Halts execution until a key is pressed cv2.destroyAllWindows()     ```

  • Writing/Saving Images: ```python

  Encodes array into target format and writes to disk

  cv2.imwrite("lucy_output.jpg", image_array)     ```

• Array Slicing, ROI, and Memory Allocation:

  • Shallow Copy vs. Deep Copy Hazard:
  • Assigning img_assigned = img creates a pointer/reference to the same underlying memory buffer. Modifying img_assigned mutates the original img array.
  • To create a completely isolated clone, explicit deep copying is required:

    img_copied = img.copy()
        ```
  - **Region of Interest (ROI) & Cropping**:
  - Extracted by slicing NumPy array coordinates `img[y1:y2, x1:x2]`:
  

  python # Extract region between row 50:200 and column 100:300 roi = image[50:200, 100:300]       ```

• Color Channel Operations:

  • Splitting and Merging:

  # Decompose into individual B, G, R 2D matrices
  b, g, r = cv2.split(image)
  
  # Recombine individual single-channel arrays into a 3D image matrix
  merged_image = cv2.merge([b, g, r])
      ```
  - **Color Space Conversions**:
  

  python

  Convert BGR to Grayscale

  gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

  Convert BGR to RGB (e.g. for Matplotlib display)

  rgb_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

  Convert BGR to HSV

  hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)     ```

• Intensity Min/Max Analytics:

  • Identifying intensity bounds is crucial for dynamic range adjustment, normalization, error detection (overexposure/underexposure), thresholding, and feature extraction.
  • Custom Red Channel Peak Extraction Code: ```python import cv2

  img = cv2.imread("sample.png") rows, cols, ncolor = img.shape red_channel_idx = 2 # BGR ordering max_red_val = 0

  for i in range(rows): for j in range(cols): val = img.item(i, j, red_channel_idx) if val > max_red_val: max_red_val = val

  print("Maximum red value in image is ", max_red_val)     ```

Point and Arithmetic Operations

• Point Operators vs. Neighborhood Operators:

  • Point Operator: Function where the output pixel intensity $O(x, y)$ depends exclusively on the input pixel intensity $I(x, y)$ at the exact same spatial coordinate (and optional global constants):     $O(x, y) = f(I(x, y))$
  • Examples: Contrast stretching, brightness shift, global thresholding, color inversion.
  • Neighborhood Operator: Function where output intensity $O(x, y)$ depends on $I(x, y)$ and the spatial neighborhood of surrounding input pixels:     $O(x, y) = f(\{I(x + k, y + l) \mid (k, l) \in N\})$
  • Examples: Image smoothing, blurring, edge filtering, spatial convolutions.

• Arithmetic Operations & Saturated Arithmetic:

  • Standard NumPy Arithmetic Hazard: Standard arithmetic on 8-bit unsigned integer arrays (uint8) causes modulo overflow wrapping ($250 + 10 = 4$; $5 - 10 = 251$).
  • Saturated Arithmetic: OpenCV enforces saturation limits to prevent wraparound artifacts:     $\text{Saturate}(x) = \begin{cases} 255 & \text{if } x > 255 \\ 0 & \text{if } x < 0 \\ x & \text{otherwise} \end{cases}$
  • OpenCV Functions: cv2.add(img1, img2) and cv2.subtract(img1, img2).

• Multiplication & Division:

  • Scaling image matrices by factors $> 1$ brightens highlights.
  • Scaling image matrices by factors $< 1$ darkens intensities.

• Weighted Image Blending (Linear Blend):

  • Combines two images linearly according to assigned weights:     $\text{dst}(x, y) = \alpha \cdot \text{img}_1(x, y) + \beta \cdot \text{img}_2(x, y) + \gamma$
  • Implementation: ```python

  Images must possess identical spatial dimensions

  resized_img2 = cv2.resize(img2, (img1.shape[1], img1.shape[0])) blended_img = cv2.addWeighted(img1, 0.6, resized_img2, 0.4, 0.0)     ```

• Thresholding: Converts grayscale images into binary format by comparing pixel intensities against a defined threshold value $T$ to separate foreground objects from background.

Spatial Padding and Boundary Handling

• Purposes of Padding:

  • Spatial Dimension Preservation: Maintains image spatial dimensions during convolutional operations in Neural Networks.
  • Data Augmentation: Allows translation, rotation, and cropping transforms without discarding border content.
  • Feature Truncation Prevention: Prevents loss of edge features situated at image borders.
  • Image Alignment: Standardizes canvas sizes when stitching images of varying dimensions.

• OpenCV Padding Syntax:

  padded_img = cv2.copyMakeBorder(src, top, bottom, left, right, borderType, value=[B, G, R])
  ```
  - Parameters `top`, `bottom`, `left`, `right`: Border thickness in pixels.

- **Border Types (`borderType`)**:
  - `cv2.BORDER_CONSTANT`: Adds a uniform colored margin specified by `value` (e.g., `value=[255, 0, 0]`).
  - `cv2.BORDER_REFLECT`: Mirror-reflects edge elements (`abcdefg` $\rightarrow$ `gfedcba`).
  - `cv2.BORDER_REFLECT_101` (or `cv2.BORDER_DEFAULT`): Mirror-reflects elements with slight edge-point adjustment.
  - `cv2.BORDER_REPLICATE`: Extends edge boundary pixels outward (`abcdefgh` $\rightarrow$ `aaaaa|abcdefgh|hhhhh`).
  - `cv2.BORDER_WRAP`: Wraps opposite edge pixels to create seamless tiling.

![Padding Modes](https://assets.knowt.com/pdf-flow-prod/45874123-3bf9-4670-b8d3-425126cd4885-figures/1.jpg)


# Drawing Operations and Video Processing

- **Drawing Shapes on Images**:
  - **Line**:
    ```python
    cv2.line(image, pt1=(50, 50), pt2=(350, 350), color=(255, 255, 255), thickness=3)
    ```
  - **Rectangle**:
    ```python
    # thickness=-1 fills the shape completely
    cv2.rectangle(image, pt1=(50, 50), pt2=(350, 300), color=(0, 255, 0), thickness=5)
    ```
  - **Circle**:
    ```python
    cv2.circle(image, center=(200, 200), radius=50, color=(0, 0, 255), thickness=-1)
    ```
  - Additional Primitive Functions: `cv2.ellipse()`, `cv2.polylines()`, `cv2.fillPoly()`, `cv2.putText()`, `cv2.arrowedLine()`, `cv2.drawMarker()`.

- **Video Stream Manipulation and FPS Processing**:
  - Video is processed as a sequential series of image frames.
  - **Webcam/File Capture Initialization**:
    ```python
    # Device index 0 captures default webcam; pass string path for video files
    cap = cv2.VideoCapture(0)
    ret, frame = cap.read()  # ret is boolean; frame is the image array
    cap.release()
    ```
  - **Custom Frame Rate (FPS) Throttle Pipeline**:
    ```python
    import cv2

    desired_fps = 10
    video_path = "sample_video.mp4"
    cap = cv2.VideoCapture(video_path)

    original_fps = int(cap.get(cv2.CAP_PROP_FPS))
    frame_interval = int(original_fps / desired_fps)
    frame_count = 0

    while True:
        ret, frame = cap.read()
        if not ret:
            break

        # Process and display frame only at target intervals
        if frame_count % frame_interval == 0:
            cv2.imshow("Frame", frame)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

        frame_count += 1

    cap.release()
    cv2.destroyAllWindows()
    ```


# Geometric Transformations and Linear Algebra Intuition

- **Mathematical Intuition of Transformations**:
  - Vector and matrix operations in spatial transformation alter coordinate vectors or modify the coordinate space itself.
  - Multiplying a coordinate vector by a transformation matrix mapped in vector space changes object position, scale, orientation, or spatial geometry.

- **Affine Transformation Definition**:
  - A linear spatial mapping that preserves straight lines and parallel line distance ratios. Object position, orientation, scale, and skew may change.

![Geometric Transformation Spectrum](https://assets.knowt.com/pdf-flow-prod/279c0977-5e88-4e03-a811-36b48bc71eb2-figures/1.jpg)

- **Matrix Formulations for Transformations**:
  - **Identity Transform**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$
  - **Scaling / Resizing**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$
  - **Reflection (Vertical Flip across Horizontal Axis)**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$
  - **Horizontal Shear**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 1 & sh_x \\ 0 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$
  - **Vertical Shear**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ sh_y & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix}$
  - **Translation / Shift (Using Homogeneous Coordinates)**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}$
  - **Pure Rotation around Origin**:
    $\begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} \cos(\theta) & -\sin(\theta) & 0 \\ \sin(\theta) & \cos(\theta) & 0 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}$

- **OpenCV 2D Rotation Matrix Formulation**:
  - Standard mathematical counterclockwise 2D rotation matrix around center $(c_x, c_y)$ with scale $s$:
    $R_{\text{math}} = \begin{bmatrix} \cos(\theta) \cdot s & \sin(\theta) \cdot s & (1 - \cos(\theta) \cdot s) \cdot c_x - \sin(\theta) \cdot s \cdot c_y \\ -\sin(\theta) \cdot s & \cos(\theta) \cdot s & \sin(\theta) \cdot s \cdot c_x + (1 - \cos(\theta) \cdot s) \cdot c_y \end{bmatrix}$
  - OpenCV defaults to a **clockwise** convention, flipping the sign of $\sin(\theta)$. The official OpenCV affine rotation matrix is:
    $R_{\text{OpenCV}} = \begin{bmatrix} \alpha & \beta & (1 - \alpha) \cdot c_x - \beta \cdot c_y \\ -\beta & \alpha & \beta \cdot c_x + (1 - \alpha) \cdot c_y \end{bmatrix}$
    where $\alpha = s \cdot \cos(\theta)$ and $\beta = s \cdot \sin(\theta)$.

- **Rotation Code Implementation**:

python import cv2

image = cv2.imread("Lucy.jpg") (h, w) = image.shape[:2] center = (w // 2, h // 2) angle = 45 # degrees scale = 1.0

# Compute 2x3 affine rotation matrix rotation_matrix = cv2.getRotationMatrix2D(center, angle, scale)

# Apply affine transform rotated_image = cv2.warpAffine(image, rotation_matrix, (w, h))   ```

• Shear Code Implementation:

  import cv2
  import numpy as np

  image = cv2.imread("image.jpg")
  (h, w) = image.shape[:2]

  shear_factor_x = 0.5
  shear_factor_y = 0.2

  shear_matrix = np.array([
      [1, shear_factor_x, 0],
      [shear_factor_y, 1, 0]
  ], dtype=np.float32)

  # Compute transformed bounding dimensions
  new_w = int(w + abs(shear_factor_x * h))
  new_h = int(h + abs(shear_factor_y * w))

  sheared_image = cv2.warpAffine(image, shear_matrix, (new_w, new_h))
  ```


# Resizing, Sampling, and Interpolation Algorithms

- **Resizing Mechanics**:
  - Adjusts image canvas spatial dimensions. Upsampling generates new pixel values; downsampling decimates pixel counts.
  - Function Signature: `cv2.resize(src, dsize, fx=0, fy=0, interpolation=cv2.INTER_LINEAR)`.

- **OpenCV Interpolation Algorithms**:
  - `cv2.INTER_NEAREST`: Nearest Neighbor interpolation.
  - `cv2.INTER_LINEAR`: Bilinear interpolation (default).
  - `cv2.INTER_CUBIC`: Bicubic interpolation over a $4 \times 4$ pixel neighborhood.
  - `cv2.INTER_AREA`: Resampling using pixel area relations. Recommended for decimation (downsampling) as it produces Moiré-free output.
  - `cv2.INTER_LANCZOS4`: Lanczos interpolation over an $8 \times 8$ pixel neighborhood.
  - `cv2.INTER_LINEAR_EXACT`: Bit-exact bilinear interpolation.
  - `cv2.INTER_NEAREST_EXACT`: Bit-exact nearest neighbor interpolation.

- **Nearest Neighbor Interpolation Mechanics**:
  - Selects the intensity value of the single spatially closest pixel from the input image and assigns it to the target pixel.
  - **Nearest Neighbor Variants**:
    - **Averaging Variant ($N=2$) Worked Example**:
      - Input pixel row: $[180, 255, 9, 56, 127, 255, 0, 73, 0]$
      - Interconstruct adjacent pixel pairs:
        - Between $180$ and $255$: $\lfloor (180 + 255) / 2 \rfloor = 217$
        - Between $255$ and $9$: $\lfloor (255 + 9) / 2 \rfloor = 132$
        - Between $9$ and $9$: $9$
        - Vertical/diagonal midpoints computed similarly: between $180$ and $56$ (below it) $\rightarrow$ $\lfloor (180 + 56) / 2 \rfloor = 118$
        - Corner midpoint $\rightarrow$ $\lfloor (217 + 118) / 2 \rfloor = 167$

- **Custom Array Upsampling Algorithm without OpenCV**:

python # Pure NumPy upsampling algorithm (factor of 2) # Row duplication half_upscaled[0:height2:2, :, :] = original_img[:, :, :] half_upscaled[1:height2:2, :, :] = original_img[:, :, :]

# Column duplication upscaled_img[:, 0:width2:2, :] = half_upscaled[:, :, :] upscaled_img[:, 1:width2:2, :] = half_upscaled[:, :, :]   ```

• Bilinear Interpolation Mathematical Formulation:

  • Computes target intensity $I_{\text{new}}$ at position $x$ between boundary points $x_1$ and $x_2$ with known intensities $I_1$ and $I_2$ (where $x_1 \le x \le x_2$):     $I_{\text{new}} = \left( \frac{x_2 - x}{x_2 - x_1} \right) \cdot I_1 + \left( \frac{x - x_1}{x_2 - x_1} \right) \cdot I_2$

• Bilinear Interpolation Worked Example:

  • Given $x_1 = 0$ with intensity $I_1 = 233$, and $x_2 = 1$ with intensity $I_2 = 200$.
  • Target spatial coordinate: $x = 0.3$.
  • Spatial distances: $x - x_1 = 0.3$, $x_2 - x_1 = 1.0$, $x_2 - x = 0.7$.
  • Weight calculation:     $I_{\text{new}} = \left( \frac{0.7}{1.0} \right) \times 233 + \left( \frac{0.3}{1.0} \right) \times 200$$I_{\text{new}} = 163.1 + 60 = 223.1$