Introduction to Computer Vision and Pattern Recognition

Visual Signal Processing in the Human Brain

• Dominance of Visual Sensory Data:

  • Total Brain Information Input: Approximately $90\%$ of all information transmitted to the human brain consists of visual data.

  • Human Communication Breakdown: Approximately $93\%$ of all human-to-human communication is conducted through visual forms.

  • Research Source Citation: Data cited from Infographic World studies (http://blog.wishpond.com/post/70300587846/10-reasons-visual-content-will-dominate-2014).

  • Practical Role: Visual status and processing heavily govern daily human activities, advanced educational learning, and professional working environments.

Foundations and Definitions of Computer Vision

• Core Definitions:

  • Automated Identification: Technology enabling systems to automatically identify and isolate objects within static images or dynamic video streams.

  • Latent Information Extraction: Techniques designed to extract underlying, unstructured data and semantic meaning from raw light stimuli.

  • Algorithmic Perception: Computational algorithms designed to mimic human sight perception and teach machines to comprehend the surrounding physical world.

  • Unsupervised Perception: Automated computation where computers learn and observe visual elements without requiring explicit manual instruction for every scenario.

  • World Perception Study: The scientific study of interpreting and understanding physical environments through visual feedback.

• Computer Vision as a Simulation of Human Sight:

  • Goal of Vision Simulation: Equipping computers with artificial "eyes" to see, identify, and categorize surroundings identically to human vision.

  • Scene-to-Concept Translation: Converting raw visual images into structured, understandable concepts such as spatial distance, structural edges, geometric orientations, and object pathways.

  • Environmental Comprehension: Teaching algorithmic software to interpret and reason about real-world scenarios through captured photographic or video data.

Historical Evolution of Computer Vision

• Decade-by-Decade Technological Progression (1970s–2020s+):

  • 1970s Foundations:

  • Digital Image Processing.

  • Blocks World environments and Line Labeling.

  • Generalized Cylinders representation.

  • Pattern Recognition fundamentals.

  • Stereo Correspondence matching.

  • Intrinsic Images extraction.

  • 1980s Algorithmic Expansion:

  • Optical Flow analysis.

  • Structure from Motion (SfM).

  • Image Pyramids multi-resolution analysis.

  • Shape recovery from shading, texture, and focus.

  • Physically-based modeling.

  • Regularization techniques.

  • Markov Random Fields (MRF).

  • 1990s Mathematical and Spatial Modeling:

  • Kalman Filters implementation.

  • 3D Range Data Processing.

  • Projective Invariants.

  • Matrix Factorization techniques.

  • Physics-based vision.

  • 2000s Feature Extraction and Segmentation:

  • Graph Cuts optimization.

  • Particle Filtering.

  • Energy-based image segmentation.

  • Face Recognition and Face Detection systems.

  • Image-Based Modeling and Rendering (IBMR).

  • Texture Synthesis and Digital Inpainting.

  • 2010s Learning and Spatial Awareness:

  • Computational Photography.

  • Feature-based recognition and Category recognition.

  • Machine Learning integration.

  • Modeling and tracking human subjects.

  • Semantic Segmentation.

  • Simultaneous Localization and Mapping (SLAM) and Visual-Inertial Odometry (VIO).

  • 2020s Modern Paradigms:

  • Deep Learning architectures.

  • Multimodal integration of Vision and Natural Language processing.

Image Interpretation and Machine Perception

• The Concept: "Every Picture Tells a Story":

  • Ultimate Goal: Writing computer algorithms capable of interpreting visual images autonomously to articulate narrative context.

  • Key Interpretive Questions Addressed by Vision Systems:

  • What is the central subject of the image?

  • What specific individual objects exist within the picture?

  • Where are these objects spatially located?

  • What structural relationships and spatial dependencies exist between objects?

  • What is the contextual relationship between individual objects and the surrounding environment?

• Internal Representation: What Computers See:

  • Absence of Direct Semantic Perception: Computers do not natively observe physical entities; they process numerical matrices reflecting light intensities.

  • Pixel Intensity Matrix: Images are represented digitally as 2D grids of intensity numbers.

  • Standard 8-bit Intensity Scaling:

  • Numerical range spans from $0$ to $255$ ($2^8 - 1 = 255$).

  • Minimum intensity value $0$ corresponds to pure black.

  • Maximum intensity value $255$ corresponds to pure white (where all 8 binary bits equal 1).

  • Brighter image regions contain higher numerical values; darker regions contain lower values.

Computer Vision vs. Human Vision

• Comparative Capabilities:

  • Performance Trade-offs: Computers do not uniformly match or beat human vision ("Yes, but mostly no!").

  • Human Cognitive Superiority ("Hard Tasks"):

  • The human brain possesses advanced contextual reasoning and inferential abilities exceeding personal computers.

  • Humans easily interpret hidden context, occluded objects, emotional nuance, and external social cues.

  • Computer Computational Superiority ("Easy Tasks"):

  • Computers execute high-speed mathematical operations, exact pixel intensity comparisons, and structured rule-based data classifications far faster than human brains.

• Shortcomings and Vulnerabilities of Human Perception:

  • Contextual Optical Illusion (Sinha and Poggio, Nature, 1996):

  • Demonstration involving 1990s US President Bill Clinton and Vice President Al Gore.

  • When presented with complete context (hairlines, suits, relative height), human observers perceive two distinct facial identities.

  • When contextual features are removed, the underlying facial features are revealed to be identical. This proves human perception relies heavily on external contextual reference points rather than pure facial geometry.

• Motion Illusions (A. Kitaoka, 2003 "Rotating Snakes"):

  • Static geometric patterns featuring repeating colored concentric circles induce false perceptions of dynamic rotation and movement in human vision.

  • Demonstrates that human optical pathways can be systematically deceived by specific spatial contrast arrangements.

State-of-the-Art Applications of Computer Vision

• 2D vs. 3D Navigation Maps:

  • 2D Mapping (e.g., Google Maps): Provides overhead geographic layouts and road navigation routes, but lacks localized 3D structural visual cues needed to recognize specific physical building facades in person.

  • 3D Mapping (e.g., Apple Maps): Uses computer vision to reconstruct three-dimensional building models (e.g., interactive 3D rotation of the Empire State Building in New York City), allowing users to visually match surrounding architecture.

• Computational Photography:

  • Portrait Mode Simulation: Smartphone software identifies human figures, body silhouettes, and facial contours, adjusting virtual camera parameters to simulate a wide lens aperture ($f$-stop control). This keeps the subject sharp while applying background blur (bokeh).

  • Astronomical Image Synthesis: In 2019, global radio-telescope arrays used computational vision algorithms to synthesize an ultra-wide aperture, capturing the first reconstructed image of a black hole.

• 3D Photo Generation (Facebook AI): Convolutional Neural Networks (CNNs) estimate depth maps from standard 2D photos to synthesize interactive 3D motion animations.

  • Biometrics and Identification:

• Identification of the "Afghan Girl":

  • Subject Sharbat Gula was photographed by Steve McCurry for the June 1985 issue of National Geographic (Vol. 167, No. 6, cover article "Along Afghanistan's War-torn Frontier").

  • Decades later, her identity was verified through vision-based iris biometrics.

  • Iris pattern structures remain stable throughout human life, serving as unique biometric markers.

• Other Biometrics: Facial scanning and fingerprint detection algorithms provide stable, real-time security identification across consumer devices and access control points.

  • Visual Search and Object Recognition:

• Google Lens: Mobile application that captures live visual input, extracts feature vectors, and matches them against database items to automatically identify real-world objects (such as specific plant species like tulips).

• Cinematic Special Effects and 3D Shape Capture:

  • Motion Picture Production (e.g., The Matrix Trilogy): Developed by ESC Entertainment, XYZRGB, and NRC. Reconstructs 3D facial profiles and spatial geometries from sequential multi-angle 2D video frames, enabling dynamic digital rendering of actor heads from any perspective.

• Sports Broadcasting Annotations:

  • Sportvision First Down Line:

  • Used in American Football broadcasts to superimpose a virtual yellow line on the field indicating the required 10-yard gain.

  • Vision algorithms differentiate grass turf from player uniforms in real time, rendering the yellow marker underneath moving players' legs rather than over their bodies.

• Interactive Gaming and Motion Tracking:

  • Microsoft Xbox Kinect: Uses dual stereo optical sensors and infrared depth cameras to track 3D body posture and physical player motion in real time without physical controllers.

• Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR):

  • Virtual Reality: Immerses users in fully synthetic virtual environments (e.g., Oculus Quest running Beat Saber).

  • Augmented / Mixed Reality: Overlays interactive digital 3D models onto real-world spatial environments (e.g., Microsoft HoloLens 2).

  • Microsoft HoloLens 2 Sensor Hardware Suite:

  • $1 \times$ 8-Megapixel RGB video camera.

  • $4 \times$ Head-tracking cameras (combining stereo and peripheral tracking).

  • $1 \times$ 1-Megapixel depth camera (supporting near-range and far-range mapping).

  • Infrared (IR) eye-tracking cameras paired with IR illumination LEDs.

  • Inertial Measurement Unit (IMU).

  • $5 \times$ Spatial microphone array.

• Advanced Robotics and Space Exploration:

  • NASA Curiosity Mars Rover:

  • Equipped with visual sensors and scientific cameras (Mastcam, ChemCam, REMS, SAM, CheMin, APXS, MAHLI, MARDI, RAD, DAN, RTG, UHF Antenna, Grand Gain Antenna).

  • Captures surrounding terrain to build local 3D spatial maps for autonomous rover navigation and obstacle avoidance across Martian pediments (e.g., Hutton drill hole, Greenheugh Pediment).

• Smart Cars and Autonomous Driving:

  • Mobileye Vision-on-a-Chip Systems (EyeQ Chip): Deployed across commercial vehicle fleets (BMW, General Motors, Volvo).

  • Multi-Camera Surround Array: Integrates forward-looking, rear-looking, and dual side-looking cameras.

  • Safety Features: Provides Advance Warning Systems (AWS), lane tracking, pedestrian protection, and automatic collision-avoidance braking.

• Aerial Drone Search and Rescue:

  • Kowloon Peak Hiker Rescue Case Study (October 31, 2018, Hong Kong):

  • A 52-year-old male hiker became lost on a cliff face along Kowloon Peak.

  • The hiker transmitted ambient scene photos but could not report exact GPS coordinates.

  • Rescue volunteers flew a vision-equipped camera drone over the region, matching landscape features to the hiker's photographs to locate him and guide ground rescue crews.

Computer Vision vs. Pattern Recognition

• Comparative Breakdown:

Practical Takeaways and Future Outlook

• Key Summary Insights:

  • Active Field Impact: Computer Vision and Pattern Recognition is an active research domain creating major societal transformations across personal, local community, municipal, and global scales.

  • Cross-Disciplinary Relevancy: Applicable across diverse academic majors and industrial sectors:

  • Urban Planning / Smart Cities: Traffic monitoring, automated surveillance, and infrastructure management.

  • Hospitality and Services: Security surveillance, occupancy tracking, and contactless client check-in.

  • Rapid Pace of Innovation: Many current state-of-the-art vision tools were developed within the past 5 years.

  • Corporate Investment: Industry leaders (including Apple, Google, Meta/Facebook, Microsoft) continue to direct resources toward computer vision R&D.

  • 5–10 Year Trajectory: Widespread deployment of novel vision-driven technologies expected over the next decade.