Filtering, Feature Detection and Extraction in Computer Vision

Image Fundamentals and Colour Spaces

Image processing and computer vision require an understanding of how images are represented digitally and how various colour models interpret visual data. Several different colour spaces exist, and while models typically consider the RGB (Red, Green, Blue) space by default, treating each channel equivalently, the relationships and transitions between spaces are often non-linear. Choosing the appropriate colour space can lead to significant improvements in computer vision tasks. The three primary spaces discussed are RGB, HSV, and CEILAB. The RGB space is a three-dimensional model comprising Red (R), Green (G), and Blue (B) channels. The HSV space captures information based on Hue (H), Saturation (S), and Value (V). The CEILAB space is particularly useful because it separates intensity from colour, consisting of an Intensity or Lightness channel ($L^<em>$) and two colour channels, $a^</em>$ and $b^*$. In practice, extracting specific channels like the Value channel in HSV or the Lightness channel in CEILAB can provide more robust data for certain algorithms than standard RGB processing.

Understanding Image Noise Models

Digital images are often corrupted by noise during acquisition or transmission, necessitating specific filtering strategies. Different sources of interference result in distinct noise models. Gaussian Noise is synonymous with sensor noise associated with natural images. Poisson Noise is characterized by its association with medical imaging and microscopy. Salt and Pepper Noise, visible as random black and white pixels, is typically caused by sensor errors and transmission faults. Using the MIT Cameraman test image resized to a $64 \times 64$ grid as a baseline, the impact of these noise types is observed numerically. For instance, a clean set of pixels might read values such as $173, 165, 164$, while Salt and Pepper noise introduces extreme values of $0$ (pepper) or $255$ (salt) into the grid.

Spatial Filtering Techniques

Filtering is a primary solution for seeing through noise and is often visualized as an operation mapping an input image $f(x,y)$ to an output image $g(x,y)$. A common approach is the sliding window method. Mean Filtering involves moving a window (e.g., a $3 \times 3$ grid) across the image and calculating the average pixel value within that window to replace the center pixel. For example, if a window contains values resulting in a mean of $95$, that value becomes the new pixel in the output image. While mean filtering can reduce noise, it often introduces considerable artifacts and fails to remove noise effectively in extreme cases. Alternatively, Median Filtering replaces the center pixel with the median value of the window. In a grid where Salt and Pepper noise is present, the median filter is highly effective at removing the extreme $0$ and $255$ values, though it may damage some fine image details.

Quantitative Performance Metrics

To objectively evaluate filtering effectiveness, quantitative comparisons are made between the original image ($u_2(x,y)$) and the filtered image ($u_1(x,y)$). The Mean Squared Error (MSE) is calculated using the formula

$MSE = \frac{\sum_{x,y} (u_1(x,y) - u_2(x,y))^2}{XY}$

where $X$ and $Y$ represent the number of columns and rows. The Peak Signal-to-Noise Ratio (PSNR) provides a logarithmic measure of image quality:

$PSNR = 10 \log_{10} \left( \frac{R^2}{MSE} \right)$

In these equations, $R = 255$ for 8-bit images and $R = 1$ for normalized images. In a test case, the Mean filter yielded an MSE of $705.17$ and a PSNR of $-28.48$, while the Median filter performed better with an MSE of $291.56$ and a PSNR of $-24.65$.

Linear Filters and Convolution Kernels

Linear filtering is fundamentally a convolution operation. A Kernel ($k(x,y)$) is a small matrix of weights used to transform the image. The general Convolution Equation is expressed as

$g(x,y) = \sum_{x'} \sum_{y'} k(x', y') u(x - x', y - y')$

Various kernels produce different visual effects. A Mean Filter kernel is a grid of uniform values (e.g., all $1/9$ for a $3 \times 3$ kernel). A Gaussian Filter utilizes weights based on the Gaussian distribution to provide smoother blurring:

$k(x,y) = \frac{1}{2\pi\sigma^2} e^{-\frac{(x-\mu_x)^2 + (y-\mu_y)^2}{2\sigma^2}}$

A typical $3 \times 3$ Gaussian kernel might use weights of $1/16$ at corners, $1/8$ at edges, and $1/4$ at the center. Motion Blur Filters utilize directional weights (e.g., a row or column of $1/3$) to simulate movement. Sharpening Filters use kernels where center values are enhanced (such as $17/9$) while surrounding values are negative ($-1/9$). A key property of these blur kernels is that their elements sum to $1$, ensuring the overall intensity of the image remains unchanged.

Edge Detection and the Sobel Operator

Edge detection is a critical step in feature extraction, identifying areas where pixel intensity changes rapidly. The Sobel Operator uses two kernels to detect horizontal and vertical edges. The horizontal Sobel kernel is

$\begin{pmatrix} 1 & 0 & -1 \ 2 & 0 & -2 \ 1 & 0 & -1 \end{pmatrix}$

while the vertical kernel is

$\begin{pmatrix} 1 & 2 & 1 \ 0 & 0 & 0 \ -1 & -2 & -1 \end{pmatrix}$

Edge detection is mathematically related to the partial derivative of the image function. The derivative measures the rate of change. Since images are discrete grids with a minimum pixel spacing of $h = 1$, the derivative is approximated as $f'(x) \approx f(x+1) - f(x)$. Using reference points $x-1$ and $x+1$ allows for a central difference approximation:

$\frac{\partial u}{\partial x} \approx \frac{u(x+1, y) - u(x-1, y)}{2}$

The gradient magnitude $G$ is then calculated by combining the results of the edge detection operators in both directions:

$G = \sqrt{(k_x * u)^2 + (k_y * u)^2}$

Other variations of edge detection kernels include the Roberts, Prewitt, and Frei-Chen operators, each utilizing different weighted grids for intensity gradient calculation.

Feature Hierarchy and Image Frequency

Features are distinctive pieces of information extracted from images and are categorized into a hierarchy. Low-level Features are directly extracted from pixels, such as edges, textures, and corners. Mid-level Features are derived from the low-level data, including object parts or contours. High-level Features consist of hierarchical representations. Image frequency measures the speed at which pixel intensity values change through space; natural images are typically a mixture of high frequencies (edges and fine detail) and low frequencies (smooth regions). This is analyzed in "frequency space" using the Fourier Transform, which decomposes an image into a Spectral Domain comprising Magnitude and Phase. Patterns and textures are encoded in the magnitude, while structure is encoded in the phase. Swapping the phase between two images demonstrates that the structural information follows the phase rather than the magnitude. Low-pass filters remove high frequencies to smooth images, while high-pass filters isolate edges. Threshold-based filters in the frequency domain can leave "ringing artifacts," which are mitigated by using Gaussian frequency filters.

Mathematical Properties and Efficiency of Convolution

Convolution exhibits several key mathematical properties: Commutativity ($k * u = u * k$), Associativity ($k * (h * u) = (k * h) * u$), and Distributivity ($(k + h) * u = k * u + h * u$). It also scales linearly with constant values ($\alpha(k * u) = (\alpha k) * u$). A vital property concerns integration: the integral of a convolved signal is the product of the integrals of the individual signals:

$\int_{-\infty}^{\infty} (k * u)(x) \,dx = \int_{-\infty}^{\infty} k(x) \,dx \int_{-\infty}^{\infty} u(x) \,dx$

The Convolution Theorem states that convolution in the spatial domain is equivalent to point-wise multiplication in the spectral (Fourier) domain:

$f * g \iff F(u,v)G(u,v)$

This theorem is crucial for computational efficiency. Performing convolution on an $M \times M$ image with an $N \times N$ kernel in the spatial domain requires $O(M^2 N^2)$ operations. In contrast, using the Fast Fourier Transform (FFT) to operate in the spectral domain requires approximately $O(N^2 \log_2 N)$ operations. For a $1024 \times 1024$ image with a $31 \times 31$ kernel, the spectral approach is roughly $50 \times$ faster than the spatial sliding window approach.

Deep Learning and Boundary Conditions

In computer vision, convolution enables translation invariance, parameter sharing, and composability. However, there is a technical distinction between mathematical convolution and the "convolution" used in deep learning. Deep learning typically employs Cross-Correlation, defined as

$k \otimes u(x,y) = \sum_{x'} \sum_{y'} k(x', y') u(x + x', y + y')$

Furthermore, deep learning convolution often omits or simplifies boundary conditions, effectively performing a convolution followed by cropping. Traditional computer vision handles image boundaries using several methods: Dirichlet Boundary Conditions (assuming values outside the domain are $0$), Periodic Boundary Conditions (assuming the image repeats), and Neumann Boundary Conditions (assuming the boundary is a reflection). Finally, while classical computer vision relies on handcrafted filters (like Sobel or Gaussian), deep learning focuses on learned filters where the kernel weights are optimized through training.