Guest Lecturer: Computer Vision: Foundations, Image Modalities, Military Applications, and Machine Learning Frameworks

Fundamentals of Computer Vision

• Computer Vision (CV) is a subfield of Artificial Intelligence (AI) focused on replicating human visual perception within machines, enabling automated systems to analyze visual scenes, extract structural details, and make autonomous decisions based on perceived visual data.
• The Human Visual Perception Analogy:
  • The human visual system perceives complex natural environments, categorizes distinct objects (such as identifying gold, apples, or oranges), and deduces contextual information using prior knowledge or real-time learning.
  • Computer vision algorithmically models this visual perception pipeline to allow hardware systems to interpret visual environments with human-like understanding.
• Primary Application Domains of Computer Vision:
  • Autonomous Systems: Self-driving vehicles (such as Tesla and Cruise) relying on multi-object detection, continuous tracking, and path perception.
  • Healthcare and Medical Imaging: Automated diagnosis through automated interpretation of X-rays and brain Magnetic Resonance Imaging (MRI) scans for disease detection.
  • Industrial and Manufacturing: Visual mitigation, visual navigation, automated inspection, and quality control.
  • Consumer Technology and Security: Facial recognition systems, smart device interactions, and household automation.

Core Computer Vision Tasks and Mechanisms

• Object Detection and Tracking:
  • Object Detection identifies, classifies, and localizes specific entities within a visual scene by generating bounding boxes around them.
  • Object Tracking maintains persistent identification across sequential video frames (for example, assigning a persistent ID such as ID 15 alongside a continuous confidence/probability score to monitor specific athletes during sports match analysis).
• Object Segmentation:
  • Unlike bounding box object detection, segmentation extracts the precise pixel-level boundaries, shapes, and instance regions of an object from an image.
  • Robotic Grasping Application: A robotic arm attempting to grip an object requires exact physical boundaries and surface measurements rather than a broad bounding box. Segmentation provides exact structural geometries enabling the robot to determine precise contact points across multiple sides (such as gripping a jug).
• Interactive Prompt-Based Segmentation:
  • Prompting Mechanisms in Vision: Analogous to Large Language Models (LLMs) like ChatGPT or Claude accepting text prompts, modern vision networks accept interactive visual inputs (such as point clicks or bounding geometry) as prompts.
  • Meta's Interactive Segmentation Architecture: Interactive models (such as Meta's Segment Anything algorithm) take user mouse clicks as prompt inputs, enabling instant and precise segmentation of target instances.
  • Circle Search: Features implemented in mobile ecosystems (such as Android or Google platforms) where drawing a circle around an object serves as a visual prompt, segmenting the targeted area and passing it to downstream search tasks.
• Pose Estimation:
  • Analyzes and estimates the spatial orientation, posture, and kinematic joint positions of articulated subjects or robotic agents.
  • Crucial in modern robotics for dynamic spatial movement, allowing robots to maintain balance and assess body configuration when standing, running, or falling.
• Depth Estimation:
  • Calculates relative three-dimensional spatial distances between the imaging sensor and various surface planes or entities within a visual scene.

Machine Learning vs. Traditional Image Processing

• Basic Image Processing Techniques:
  • Rely on fixed mathematical transformations and hand-crafted feature extractors.
  • Function with low training data requirements, but experience severe performance ceilings in complex visual environments.
• Deep Learning and Machine Learning Models:
  • Model performance increases statistically as both training data volume and network architectural complexity increase.
  • Requires tailored dataset curation, training pipelines, and domain-specific model backbones to maximize accuracy across complex vision tasks.

Modalities of Visual Data

• Standard RGB (Color) Imaging:
  • Represents standard digital photography produced by consumer smartphones and laptops.
  • Captures visible light across specific spectral bands typically ranging from $300\text{ nm}$ to $700\text{ nm}$.
  • Uncompressed frame sizes generally range from $3\text{ MB}$ to $5\text{ MB}$.
• Synthetic Aperture Radar (SAR):
  • Specialized active radar imaging modality requiring custom radar hardware; cannot be captured via standard digital phone cameras.
  • Penetrates adverse atmospheric conditions (such as dense rain, fog, cloud cover, and airborne environmental hazards) that completely degrade standard optical sensors.
• Infrared (IR) Imaging:
  • Detects thermal signatures and infrared radiation emitted by objects and living organisms.
  • Converts heat emissions into visual contrast, illuminating warm targets (such as locating lost individuals in dense forests or woods).
  • Widely deployed in low-light environments, tactical nighttime military operations, and extreme combat defect scenarios.
• Hyperspectral Imaging (HSI):
  • Collects contiguous spectral data across vast wavelength ranges (extending from $300\text{ nm}$ up to $2000\text{ nm}$ or $2500\text{ nm}$).
  • Generates multi-dimensional spectral data cubes, with individual frame file sizes reaching several gigabytes ($\text{GB}$) due to continuous physics-based spectral recording.
  • Applied extensively across industrial quality control, agricultural monitoring, and automated food safety inspection.

Research Project 1: Military Equipment Impact Classifier in Stressed Environments

• Tactical Objective:
  • Develop robust neural network classifiers to accurately categorize military vehicle images in battlefield conditions, assisting army command decisions and preventing friendly fire incidents (preventing troops from destroying friendly units).
• Stressed Battlefield Image Conditions:
  • Environmental Obscurants: Heavy atmospheric fog, severe rainfall, mud, dust, and visual distortion.
  • Intentional Camouflage: Vehicles actively blended into surrounding terrain textures to conceal structural boundaries from optical detection.
  • Visual Noise: Coloration loss, partial occlusions, and severe visual artifacts.
• Initial Dual-Stream Pipeline Architecture:
  • Input images undergo processing through a auxiliary segmentation network to extract structural shape maps and boundary details.
  • A primary machine learning backbone extracts visual appearance features from the raw input images.
  • Visual feature maps and structural shape features are merged, passing fused feature representations into the final classification layer.
• Multi-Task Feature Fusion Expansion:
  • Integrated Object Detection: Sub-components (such as gun barrels, tread tracks, and armored turrets) are detected to extract localized domain signatures.
  • Full Instance Segmentation: Background and foreground segments are extracted to capture environmental and structural context.
  • Fused Multi-Task Vector: Visual representations, explicit sub-component detection outputs, and segmentation maps are concatenated before entering the final military vehicle classifier.
• Model Evolution and Framework Selection:
  • Mask R-CNN: Deployed in earlier project iterations (around 2020) as a specialized instance segmentation framework.
  • YOLO Ecosystem (YOLOv8): Updated pipelines transitioned to YOLOv8 (You Only Look Once version 8), which offers superior detection accuracy, processing speed, and optimized network architecture compared to older Mask R-CNN models.

Research Project 1 (Continued) & Project 2: Thermal IR Classifier and Hierarchical Architecture

• Thermal IR Classification Challenges:
  • IR imagery lacks multi-channel visual color detail present in standard RGB images.
  • Stressed thermal signatures frequently present missing vehicle components or thermal blending, where target vehicle surface temperatures match ambient terrain background temperatures (e.g., a car body or animal matching surrounding thermal radiation).
• Loss Function Optimization for Background Blending:
  • Standard cross-entropy loss causes attention heatmaps to scatter across irrelevant background surroundings.
  • By implementing loss function modifications, model attention maps are constrained directly onto the vehicle body, eliminating background interference and maintaining high classification accuracy despite surface temperature blending.
• Project 2: Hierarchical Military Image Classification:
  • Flaw of Standard Flat Classifiers: Standard classifiers penalize all prediction errors equally, failing to distinguish between minor misclassifications and catastrophic tactical errors.
  • Example Structural Hierarchy:
  • Top Level: Military Vehicle
  • Mid Level: Tank Family vs. Armored Wheeled Vehicle Family
  • Fine-Grained Level: Abrams Tank vs. Bradley Fighting Vehicle vs. MRAP / AMPV
  • Severity Comparison: Misclassifying an Abrams tank as a Bradley fighting vehicle keeps the prediction within the correct parent family ("Tank"). Misclassifying an Abrams tank as an MRAP (wheeled vehicle) jumps parent families, causing a severe tactical error.
  • Hierarchical Loss Mitigation: Enforces taxonomy-aware classification. If the classifier fails fine-grained target resolution, it safely defaults to the correct parent taxo-level (e.g., classifying the image as a "Tank"), ensuring reliable operational output in real-world deployments.

Research Project 3: Hyperspectral Imaging for Foreign Material Detection

• Industrial Application in Food Safety:
  • Deployed by organizations like the USDA and commercial food manufacturing facilities to inspect raw poultry (chicken meat) on processing lines prior to commercial distribution.
  • Identifies tiny foreign matter, non-organic contaminants, and microplastics embedded within poultry meat that are completely invisible under standard optical RGB lighting.
• Technical Detection Pipeline:
  • High-dimensional hyperspectral line-scan data cubes ($300\text{ nm}$ to $2500\text{ nm}$) are captured as chicken meat passes along automated conveyers.
  • The spatial-spectral data cube enters neural network processing modules, generating localized embeddings and spectral queries.
  • Output segment maps flag non-organic contaminants (highlighted via distinct green mask overlays), enabling automated removal of foreign contaminants prior to supermarket delivery.
• Foundation Models in HSI: Ongoing research focuses on constructing generalized visual foundation models engineered specifically to process multi-gigabyte hyperspectral data cubes.

Practical Implementation Frameworks and Learning Resources

• Software Frameworks for Machine Learning and AI:
  • PyTorch: Created by Meta; primary Python-based deep learning framework utilized throughout the lab's research for building model backbones, customizing loss functions, and cascading pipelines.
  • TensorFlow: Created by Google; widely adopted Python-based framework for machine learning development and industrial model deployment.
• Educational Platforms and Repositories:
  • Ultralytics: Comprehensive platform hosting optimized computer vision architectures, including the YOLO algorithm suite (e.g., YOLOv8).
  • Coursera: Online educational platform offering foundational and advanced coursework in artificial intelligence, machine learning, and computer vision.
  • Hugging Face: Open platform for downloading pre-trained deep learning models, fine-tuning custom datasets, and sharing machine learning repositories.

Questions & Discussion

• Question: What is the technical distinction between prompt-based segmentation and standard bounding box generation, and are manual bounding box annotations considered prompts?
  • Response: A bounding box produced by a video detector represents a finalized model prediction or decision output. A prompt represents an input vector provided to the network to specify target region extraction (such as point clicks or user shapes). When manually annotating dataset images, bounding box inputs can technically act as prompts depending on architectural design (such as Meta's prompt-driven click segmentation network).
• Question: What prompted the transition from Mask R-CNN to YOLO architectures across project iterations?
  • Response: Mask R-CNN served as the primary specialized segmentation framework during earlier research phases (circa 2020). As deep learning architecture evolved, modern models like YOLOv8 emerged, providing superior processing speeds, higher detection accuracy, and better architectural efficiency.
• Question: At what research stages is the PyTorch framework integrated into model development?
  • Response: PyTorch is integrated from the initial project implementation phase. Its modular Python structure enables researchers to write custom network blocks and cascade multiple model outputs—such as piping explicit spatial feature masks generated by a segmentation model directly as input tensors into a secondary downstream classification model.