Computer Vision Lecture Notes

Introduction to Computer Vision

• This week's topic: Computer Vision.
• Builds upon classifiers, linear regression, perceptrons, and neural networks.
• The first lecture introduces seemingly overcomplicated concepts, which will be integrated into modern deep convolutional neural networks in the second lecture.

The Challenge of Computer Vision

• Computer vision is more complex than it appears.
• Humans use prior knowledge and context (e.g., the number of fingers, the structure of a human body) to interpret images.
• Computer vision models lack this inherent understanding and rely solely on the provided image and training data.
• Object occlusion: A significant challenge where parts of objects are hidden.
• The task involves inferring high-level information from low-level image data.

Applications of Computer Vision

• Constructing higher-dimensional understanding.
• Object detection and classification.
• Image segmentation: Identifying which pixels belong to specific objects (e.g., bicycle, person).
• Meta's Segment Anything model: An algorithm for effective image segmentation.
• Levels of Tasks:
  • Is there a car (yes/no)?
  • Where is the car?
  • Drawing pixels around the car (image segmentation).
  • Identifying car features.
• Applications range from Earth-based to extraterrestrial (e.g., rovers on Mars).
• Rovers use computer vision for panorama stitching and assessing terrain (e.g., cliffs) to understand 3D space.
• Computer vision can be integrated with other data sources such as GPS and accelerometers.

Image Data as Matrices

• Image data is represented as matrices.
  • Black and white images: Single matrix with values from 0 to 255 (or normalized between 0 and 1).
  • Color images: Three matrices representing red, green, and blue channels.
• Matrix operations are used for image processing (e.g., dot products, matrix multiplication).
• Hardware and software efficiently handle matrix operations.

Bio-Inspired Computer Vision

• Early computer vision (pre-1980s): Focused on detecting edges, corners, and lines using custom algorithms.
• Inspiration from the Brain:
  • The brain's visual cortex processes information hierarchically through layers.
  • Layer 1: Processes photons.
  • Layer 2: Processes the output of Layer 1 (groups of photons).
  • Subsequent layers process increasingly complex features.
  • A single neuron in Layer 6 might activate for a specific face.
• Deep neural networks aim to mimic this hierarchical processing.
• Nature-inspired computation: Algorithms that emulate brain functions.

Perceptrons and Neurons

• Perceptron (neuron): Takes n-dimensional data as input (k-dimensional in the transcript example).
• Each feature is multiplied by a weight, and a bias is added.
• The result is passed through a nonlinear function (e.g., step function).
• Output: 0 or 1 (if a step function is used).
• Equation: $output = f(\sum<em>{i=1}^{k} w</em>i * x<em>i + b)$, where $f$ is the nonlinear function, $w</em>i$ are the weights, $x_i$ are the inputs, and $b$ is the bias.

Multilayer Perceptrons

• Simplest form of deep neural network.
• Composed of multiple perceptrons arranged in layers.
• Each perceptron has its own weights and biases.
• Example:
  • Input layer: 6-dimensional data.
  • Hidden layers: Compress data to 4 dimensions, then 3 dimensions.
  • Output layer: Single number (0 or 1) for classification.
• The number of layers and neurons per layer are hyperparameters.
• Deeper networks generally perform better than wider networks due to better feature abstraction.
• Deeper networks can identify composite features (e.g., eyes, smiles) in faces.

Challenges with Multilayer Perceptrons for Images

• Example Problem: Detecting 'X' or 'O' in a 6x6 image.
• Representing the image as a 36-dimensional vector.
• Using a multilayer perceptron to classify the image.
• Issues:
  • Doesn't scale well: Large images require many parameters.
  • Not robust: shifting the image changes the input vector significantly.
  • Ignores the spatial correlation between pixels.
• The vertical representation of the image loses the inherent image structure.
• Brute force approaches do not take advantage of the fact that the data is an image.

Inspiration from Neuroscience: Feature Detection

• Experiments in 1959 involving cats and electrodes revealed neurons that respond to specific line orientations.
• Artificial neurons could be made to detect general patterns rather than individual pixels.
• This led to the concept of filters: Small patterns (e.g., 3x3) that scan the image to find specific features.
• Filters don't just detect if pixels are on/off, but look for patters of pixels like diagonal or horizontal lines.

Convolutional Filters (Kernels)

• A new type of neuron with weights arranged on a grid.

• Instead of randomly spread weights, there are nine weights arranged on a 3x3 grid.

• Scans over the image, learning weights on that square.

• Dot product: Used to see if the image matches the filter's learned weights.

• Formula: $Data * Weights + Bias$

• Filter/Kernel: These terms are interchangable.

• A new table is created to store the filter output: Feature Map.

Convolution Operation

• Scanning the image using the filter pixel by pixel.
• The filter remains consistent to identify its learned pattern throughout the image.
• Activation: Occurs when the filter finds a match. The dot product is performed plus bias, and put it into the feature matrix.
• A nonlinear function is then applied.
• Positive Value: If the filter finds a perfect match on the image
• The feature map shows where that specific filter/pattern appears in the image.

Feature Maps

• The feature map size will depend on the image size and filter size.
• For a 6x6 image with a 3x3 filter, the feature map will be 4x4.
• Algorithm learning: The filter updates itself and looks for desired patterns such as diagonals.
• Feature map values are positive where filter patterns are found and negative where they're absent.

Deep Convolutional Neural Networks

• Built upon the convolution step.
• Developed after 1980 with more architectures, algorithms, data, and resources.
• Around the 2000s, it was rebranded as "deep learning."

Neural Network Architectures

• Initial Image: Images are often 64 by 64 with red, green, and blue (three channel's).

• A 5x5 filter is used, resulting in a 60x60 feature map, (64 feature maps total, indicating 64 filters were used).

• From the input layer to the first layer, the neuron must learn ~1,600 parameters.

• A cube of features is produced, either by 60x60x64 or a 64 feature map.

• Blocks: These represent one filter from the previous block, and a resulting 24x24x256 feature map.

• From the previous step, there's a filter with 256 Filters.

• Additional functions, such as dimensionality reduction, help further this process.

• Images become quite small at images of 12x12, however there are 192 of those types of images.

• In summary, the smaller images represent filters made up of other filters and patterns.

Fully Connected Layers and Softmax

• Images are flattened similar to data in linear regression.
• Softmax is used to classify data into, for example, 10 options/features.
• Deep Convolutional Neural Networks: a buildup of smaller concepts, as the lecture explained, with the right side having multilayer perceptron type components.

Computer Vision Papers

• Typically you will see versions of the same concept: A cube or diagram representing information flow.
• Architecture: How to organize the images and layers in the neural network.
• Example: AlexNet, which won a competition (detecting 100 or 1,000 classes of objects). The overall accuracy and precision was low before it was developed, as prior year's winners would only come in slightly higher than others.
• After AlexNet's entrance, AI became difficult to follow as the logic for things became too complicated. An oddity noted was that it ran with two GPUs and split training into columns, and then merged at the end. These algorithmic reasons don't matter, as even the creator thought to build things differently if he had more GPUs.
• Ultimately use every tool and data available to get your own network built!

Transfer Learning and Data Augmentation

• Often, there's pre-trained algorithms that can be fine-tuned (transfer learning for reuse from other sources). In the presented research, a model that detected 20 objects was instead trained for also detecting trees and rocks by relabeling data as such and feeding it back to the network.

Generative AI and Limitations of Deep Learning

• Even this type of algorithm can be use in tandem with other tools: AlphaGo used it to see the Game Board, even if it wasn't the core algorithm.

• With image classifications with AlexNet, we often don't need to train it or we can build off a pre-trained algorithm trained by others.

• Deep learning models are very hungry for data, millions if not more.

• Data augmentation help creates larger data sets. By slightly altering data set, and adding it, you can artificially create new data points. For example, images of people can be flipped horizontally to double the data.

Computing Resources and Hardware

• Good machine learning models use GPUs or cloud computing.
• Google is developing TPUs (Tensor Processing Units) and there are many companies that sell specific hardwares for AI.
• Hardware limitation is a big issue. The Boss man for Google literally are trying to bribe Nvidia to hold back some of their GPUs so they can develop bigger AI Machines.
  *** This limit has created a push to make smaller and more efficient programs, potentially used on distributed networks.
• These networks are huge, a lot of parameters. How does one know how to choose how many layers, or the best learning method? Some models get to a trillion parameters eventually.

Hyperparameters and Limitations of Big Models

• Often someone makes as smaller version that may be just as good.
• Example: A model with a trillion parameters ends up being made with a billion, but with only 1% accuracy loss.
• Deep learning models don't represent when they do not know the answer.
• Deep learning algorithms work on probabilities, and often times they are too similar.
• In some contexts, it is good to have some kind on uncertainty estimation. We want the machine to admit if he doesn't know the answer.
• We are still struggling in finding, how can we implement models in some context to represent the data better?
• Most of the deep learning work is just, making things bigger, and in the end does not seem useful.
• Do we wanted to spend 40,000 to improve the models for just a little amount? Does it make sense for business case?