Multilevel Computational Theories of Perception and Cognition: From Inverse Graphics to Mental Simulation

Bridging Perception and Cognition via Multi-Level Computational Models

• Human perception does not perceive a jumble of light or colorful patterns but instead identifies rich, discrete structures for planning and thinking.

• These structures include:   - Objects with $3D$ shapes and specific physical properties.   - Scenes with navigable surfaces.   - Events containing temporarily demarcated dynamics.   - Agents with discernible goals and intentions.

• The central question of research is how perception transforms raw sense data from the physical environment into manageable concepts like objects and people.

• Multiple disciplines study this transition:   - Psychology: Researching the experience of perception and visual cognition.   - Cognitive Science: Analyzing cognitive, algorithmic, and representational bases.   - Neuroscience: Grounding perception in cortical signals.   - AI and Data Science: Realizing perception through computation and machines.

• The research presenter adopts a "Middle to Out" approach, starting with cognition and cognitive theories to build multi-level computational models.

• These models are designed to be integrated, explaining empirical measurements across multiple scales, from psychophysics to single-cell electrophysiology, using the formal language of cognitive theories.

The Causal Generative Model Framework

• The foundation of this cognitive theory is causal generative models, rooted in mathematical representation theory and computer science.

• Metaphor of Childhood Height Measurement:   - A graphite strike on a wall represents a child's height.   - Property 1 (Structure Preserving): The ordering of marks on the wall preserves the physical ordering of the child's actual height over time.   - Property 2 (Behaviorally Efficacious): The representation enables high-reward actions, such as selecting appropriate clothing sizes or tracking growth across time.

• Causal Generative Models in Perception:   - These models implement abstractions of physical causes (light, materials, objects).   - They describe how objects move and react to external forces and project onto human retinas as images.   - They serve as the "glue of intelligence," acting as the target for sensory processing during perception and the mechanism for mental simulation and planning.

• The objective is to link these theories to high-resolution neural data to explain mechanisms in functionally interpretable ways and to utilize the psychophysics toolbox (e.g., the flicker Paradigm).

The Inverse Graphics Hypothesis and Neural Mapping

• The Inverse Graphics Hypothesis posits that biological vision has internalized external computer graphics-like generative processes, but operates in the reverse order.

• Implementation Method:   - An inference network (Standard Deep Convolutional Neural Network or DCNN) is trained end-to-end to map the outputs of a generative model (images) back to its inputs (seen-level causes).   - Success is defined by the network's ability to recover fine details of $3D$ shape, texture, and expression in both synthetic and real-world images.

• Hierarchy of Representations:   - The network progresses from a $2D$ retinal image through a $2.5D$ stage (representing surface properties and hidden non-linearities).   - It ultimately culminates in late-hierarchy layers representing $3D$ scene-level properties.   - Empirical tests show that in the bottom-up hierarchy of the DCNN, the $2.5D$ like stage is reached earlier than the $3D$ like stage.

Case Study: The Macaque Face Patch System

• Face perception provides more than identity; it allows for the perception of momentarily detailed $3D$ shape, texture, expression, and health cues.

• Human psychophysical evidence shows people can match visually presented faces to $3D$ printed sculptures through touch alone.

• Neural Regions Studied: Winrich Freiwald and Doris Tsao identified a subcircuit of face-selective regions in the macaque inferior temporal (IT) cortex: ML/MF (posterior), AL (middle), and AM (anterior).

• Experimental Data:   - Single-cell recordings from hundreds of neurons per site.   - Stimuli: $175$ face images ($25$ identities $\times$ $7$ poses).   - Analysis: Representational Similarity Analysis (RSA) using normalized dot products of brain response vectors to create similarity matrices.

• Transformation of Neural Representations:   - ML/MF: Exhibits view specificity (physically close poses are represented similarly).   - AL: Exhibits mirror symmetry (left and right profiles are more similar to each other than to other intermediate poses).   - AM: Exhibits view-robust identity coding (neurons encode identity independently of the viewing angle, visualized as paradiagonal stripes in the matrix).

• Model Comparisons:   - Inverse Graphics Model: High fidelity match to the neural transformations, including the mirror symmetry stage.   - VGG Face Network: Quantitative and qualitative poor fit; the model skips the intermediate mirror symmetry stage and moves directly to view invariance.

Generalization: Visual Processing of Bodies in the IT Cortex

• Bodies represent a rich source of visual information regarding posture, pose, and heading direction.

• Neural Regions Studied: MSP (more posterior) and ASP (more anterior) patches in the macaque body processing network.

• Body Inverse Graphics Model:   - Description: Neutral pose $3D$ shape and posture are projected to an articulated $2.5D$ surface and then to $2D$ image plane features.   - Training: End-to-end DCNN mapping images to latent variables of the generative model.

• Comparative Analysis: The model was pitted against unsupervised models (Sim clear, Mas Cod) and supervised hierarchical models.

• Results: Inverse Graphics correctly predicted transformations between MSP and ASP, whereas alternative models failed to capture the intermediate representations.

• Conclusion: Inverse Graphics may represent a canonical computation organizing networks in the inferior temporal cortex.

Physics-Based Representations in Soft Object Perception

• Challenging the Bottom-Up Perspective: Visual perception is not solely based on light bouncing off an object; humans can perceive shapes of entirely occluded objects (e.g., a chair shape visible only through the way a cloth drapes over it).

• Computational Approaches for Soft Objects:   - Deep Neural Network (DNN): Trained to map cloth animations to latent physical properties like stiffness and mass.   - Woven Model (Intuitive Physics): A probabilistic inference model using a causal generative model of cloth physics. It defines prior distributions over forces/properties by simulating a $Spring-Mass System$.

• Prediction via Woven: The model conditions the generative model on sensory observations to perform inference (resembling a Kalman filter process).

Neural Dissociation in Cloth Perception (fMRI)

• Study Stimuli: $8$ animations across $4$ scenarios ($Ramp$, $Drape$, $Rotate$, and $Wind$) involving two cloth types ($Soft$ like silk and $Stiff$ like denim).

• Methodology: Participants viewed animations during fMRI without an explicit physics task (spontaneous visual processing).

• Regions of Interest (ROI):   - Physics ROI: Frontal-parietal regions activated during explicit physical reasoning.   - V1: Early visual processing stage.

• Analysis Results:   - Linear classifiers decoded stiffness information from both the Physics ROI and V1 across all scenarios.   - A double dissociation was discovered: The Physics ROI matched the Woven model (physics-based), while V1 matched the image-feature-based DNN.

• Implications: The brain utilizes multiple computational codes simultaneously to solve visual perception problems.

Towards Neural Mechanisms of Mental Simulation

• Future direction: Programming causal generative models directly into neural networks without end-to-end training.

• Methodology (Danny Basset and Jason Kim): Symbolic physics systems and ordinary differential equations (ODEs) can be expressed as circuits of dynamical objects and programmed into the weights/connectivity of a Recurrent Neural Network (RNN).

• Application: Modeling macaque prefrontal cortex dynamics while playing the video game of $Pong$.

• Findings: The programmed RNNs showed consistency with actual neural data, suggesting a new path for uncovering the neural mechanisms of mental simulation.

Questions & Discussion

• Question: How do you design your forward graphics model? Are they designed by the researchers or taken from existing algorithms?

• Response: The forward models are part of the framework for structure-preserving mappings. Both the face and body models used in studies are based on empirical data, such as human scans. While the learning of these models in biological systems is an open question, the current models are not arbitrary; they are structural matches for physical bodies/faces. The speaker notes that under this theory, different computer graphics implementations should yield similar results as long as they are homomorphisms of each other.