Cognition Lectures 10 and 11- Semantics + Large Models

Semantic Categories

Hierarchical tree of concepts

Semantic Categories

• Showing people movies (a dynamic stimulus)

• Can assign everything in a frame to a category

Semantic Categories (cont)

Estimating the semantic tuning of each voxel in the brain

Key takeaway

• Category information is widely distributed across many areas of the cortex

• Stimulus-selective regions (e.g. PPA) are “selectivity peaks” in this distribution

Computational Modeling

Approximating the mind with math

• Key idea: emulating the code and operations of mental representation in an alternate, artificial medium

Perceptron

The first age of the neural network

Neocognitron

Hierarchical network structure, similar to human brain

Modern Neural Networks (2012-now)

Larger version of earlier networks (more layers ~ “deep”)

• The key breakthrough that enabled modern neural networks are: more layers and more training data

• First breakthrough happened for visual categorization, the most explored domain up to the 2010s

Towards Human-Scale Categorization

The first neural network that could distinguish between thousands of items

Usefulness of Deep Neural Networks (DNNs)

Good prediction for human behavior 

• Greatly improved ability to predict performance, judgements, and error patterns compared to e.g. HMAX

Representational Similarity Analysis (RSA)

Quantifying the joint similarity structure of a set of items

*silly picture on slides

Usefulness of Deep Neural Networks (DNNs)

Good model for neural representations across (visual) cortex

• Greatly improved ability to predict performance, judgments, & error patterns compared to e.g., HMAX

• They start capturing things in meaning space rather than just what they look like

Usefulness of Modern Neural Networks

• A lot of these networks are similar to the endpoint of computing in the brain

• We can build models that predict what humans are doing

Are DNNs Good Models for the Visual System

Sort of, maybe?

• Large improvements in both categorization performance and prediction IT/LO neural responses

Categories as Collections of Features

Slow vs. fast, big vs. small, furry vs. smooth, etc

How Categorization Works: A Theory

Categories as multidimensional Twizzlers

• When a image hits our retina it doesn’t look like it should

• Visual system turns A into B

Are DNNs Good Models for the Visual System?

Increasingly yes; potentially because they untangle manifolds

• High categorization performance and good precision of IT/LO neural responses

Emulating Complex Behavior

DeepFakes; Mimicking appearance, voice, mannerisms, etc

• AI doesn't care about solving the problem you give them, they just care about giving the answer that they were trained to give

Word Embedding Models

Capturing semantic relationships between concepts

Syntactic context

• “The bear was no larger than the bull”

• “Told him to ride the bull, but not the bear”

• “Traded two bull pelts for one bear rug”

• 
  

Goal

• Given an input word (“bear”)

• Predict next word in the sentence

Result

• Words that co-occur often are encoded

• Similarly in hidden layer

Emulating Complex Cognitive Behavior

ChatGPT/Conversational Agents: Talking to the void (and it talks back)

• Realistic conversations

• Breadth of fact-based “knowledge”

• Can convincingly sound like a real person

  • I.e. passes the “Turing test”

• Can tell you how to build a nuclear bomb