Comprehensive Cognitive Science and Computation Midterm Review Notes

Foundations of Cognitive Science and Constructive Perception

• Interdisciplinary Nature of Cognitive Science:

  • Cognitive Science is an interdisciplinary field dedicated to understanding the mind, intelligence, and cognitive processes by unifying five distinct disciplines:

    • Psychology: Investigates what minds do (behavior, perception, memory, and cognitive action).

    • Philosophy: Investigates what minds are (the nature of mental states, consciousness, and representation).

    • Neuroscience: Investigates how physical brains work (biological implementation and neural architecture).

    • Computer Science: Constructs computational models of what minds do (algorithms, AI, and execution systems).

    • Linguistics: Investigates how minds communicate (structure, acquisition, and processing of language).

• Core Computational Assumption:

  • The cognitive approach explicitly treats the human mind as an information-processing and computational system.

  • The system receives inputs from the environment, stores and retrieves information, transforms internal representations using systematic processes, and produces behavioral outputs.

• Algorithms and Cognitive Constraints:

  • Definition of Algorithm: A formal, step-by-step procedure for generating outputs from inputs through operations performed on intermediate representations.

  • Cognitive Constraints: Human cognition is strictly constrained by physical and biological limitations, such as limited memory capacity and processing time limits.

  • Multiplication Example: Cognition requires efficient algorithmic procedures rather than static memorization of every possible arithmetic combination. The human mind cannot simply store a infinite lookup table for multiplication; it must execute procedural transformations over intermediate representations.

• Constructive Perception:

  • Rejection of the Mirroring Paradigm: Perception is not a direct mirroring or passive copying of objective reality ($\text{World} \rightarrow \text{Mind Copies It}$).

  • Constructive Model: Perceptual experience is actively constructed by combining raw sensory inputs with internal assumptions, prior knowledge, and heuristics ($\text{Sensory Input} + \text{Assumptions/Knowledge} \rightarrow \text{Perceptual Experience}$).

  • Empirical Demonstrations of Constructive Perception:

    • Kanizsa Triangle / Ehrenstein Illusion: The visual system projects phantom boundaries, illusory edges, and localized brightness enhancements where no physical light intensity changes exist in the stimulus.

    • Ponzo Illusion: Depth and visual context alter size judgments; identical parallel line segments appear unequal in length because the mind automatically applies distance-scaling heuristics.

    • Ames Room: Inbuilt assumptions regarding standard rectangular room geometry cause observers to misjudge the relative physical size of humans standing in different corners of an irregularly shaped trapezoidal room.

• Cognitive Heuristics and Perceptual Distortions:

  • Availability Heuristic: Individuals estimate the frequency, probability, or prevalence of events based on how easily specific instances are retrieved from memory, rather than objective statistical frequency.

    • Example: People often believe more English words start with the letter "K" than have "K" as their third letter because memory indexing makes initial letters significantly easier to retrieve.

  • Oddball Effect: Novel, highly unexpected, or salient stimuli are perceived as having a longer duration in time than familiar, repetitive stimuli. This perceptual distortion occurs because novel stimuli command heightened attentional allocation, increasing internal clock activity.

Marr's Tri-Level Hypothesis and Levels of Explanation

• Limits of Physical Reductionism:

  • David Marr demonstrated that analyzing physical biological hardware alone (such as recording individual neuronal activity) is fundamentally insufficient to explain cognitive processes.

  • Bird Flight Analogy: Understanding how birds fly requires a theoretical grasp of aerodynamics and fluid dynamics; examining physical feathers under a microscope is not enough. Similarly, explaining visual perception requires understanding the computational problems vision solves, not merely recording electrical spikes in neurons.

  • Different scientific questions necessitate distinct levels of abstraction.

• Marr's Three Levels of Explanation:

*   **1. Computational Theory Level (Shortcut: WHAT/WHY):**
    *   Defines what the system's goals and properties are, and why this specific computation is performed.
    *   Focuses on the core objective of the task and the logical constraints governing the domain.
*   **2. Representation & Algorithm Level (Shortcut: HOW):**
    *   Specifies what information is used, what format represents that information, and what step-by-step rules or algorithms transform inputs to outputs.
*   **3. Hardware Implementation Level (Shortcut: PHYSICAL):**
    *   Identifies what physical objects, biological structures, or hardware components physically perform the computation.

• Applied Examples of Marr's Levels:

  • Computational Level: "The system must compute the shortest navigational path between two geographical points."

  • Representation & Algorithm Level: "Spatial locations are formatted as numerical coordinates and transformed using Dijkstra's graph search algorithm."

  • Hardware Implementation Level: "Biological neurons in the hippocampus encode spatial coordinates via action potentials."

• Interdependence and Autonomy of Levels:

  • Marr's three levels are hierarchically linked but partially independent.

  • Knowing the physical hardware implementation (e.g., biological neurons or silicon transistors) does not automatically reveal the algorithm being executed or the overarching computational goal.

• Representational Selection and Trade-offs:

  • The same underlying entity can be formatted into distinct representational systems. For example, the integer $37$ can be formatted in binary notation as $100101$ or in Roman numerals as $\text{XXXVII}$.

  • The choice of representation determines which information is explicit and easily accessible, as well as which computational operations are simple or difficult to execute.

• Application to Vision:

  • Explaining vision requires specifying:

    1. What problem vision solves (e.g., deriving 3D spatial properties from 2D retinal projections).

    2. What representations and algorithms solve it (e.g., edge detection algorithms, depth mapping).

    3. How those processes are physically realized (e.g., photoreceptors, visual cortex V1).

Classical Computation and Computational Theory of Mind

• Historical Evolution of Computing Devices:

  • Antikythera Mechanism: An ancient mechanical calculator built to predict astronomical positions and eclipses; it was a specialized, non-programmable physical mechanism rather than a general-purpose computer.

  • Babbage Difference Engine: Designed by Charles Babbage as a specialized mechanical calculator to compute polynomial functions.

  • Babbage Analytical Engine: Proposed by Babbage as a general-purpose, mechanical, punch-card-programmable computer.

  • Ada Lovelace: Authored what is recognized as the first computer program—an explicit algorithm designed for the Analytical Engine to compute Bernoulli numbers. Lovelace realized that general-purpose machines could manipulate non-numerical symbols representing abstract relationships.

• Classical Computational Theory of Mind (CCTM):

  • Core Thesis: The human mind literally is a computational system.

  • Memory representations consist of discrete physical symbols.

  • Algorithms consist of sequences of rule-governed step-by-step operations over those symbols.

  • CCTM does not merely claim the mind is like a computer; it asserts that the mind is an instantiated symbol-processing computer.

• Core Properties of Symbolic Systems:

  • Systematicity (Rearrange Components): The systemic capacity to process or represent specific thoughts implies the automatic ability to represent structurally related thoughts using rearranged components. For example, any system capable of representing "John loves Mary" possesses the structural capacity to represent "Mary loves John."

  • Productivity (Keep Combining Components): Symbolic architectures use recursive structural rules to combine a finite set of primitive components into an infinite number of complex representational expressions.

• Historical Milestones in Symbolic AI:

  • Logic Theorist (Allen Newell & Herbert Simon, 1956): Used discrete symbols representing logical statements and explicit derivation rules to generate formal proofs. It proved that symbolic computation can perform non-numerical, formal deductive reasoning.

  • ELIZA (Joseph Weizenbaum): A conversational program relying on keyword matching, syntactic decomposition rules, and reassembly templates to simulate a Rogerian psychotherapist. ELIZA generated outputs that appeared intelligent without true comprehension, proving that human-like behavior does not imply human-like underlying mechanisms or understanding.

• ACT-R Cognitive Architecture:

  • Declarative Memory: Stores explicit facts and general knowledge (e.g., knowing that $3 + 4 = 7$).

  • Procedural Memory: Stores action rules and step-by-step execution procedures formatted as production rules (e.g., $\text{IF } X \text{ occurs} \rightarrow \text{THEN perform } Y$).

• Turing Machines and Universal Computation:

  • Components of a Turing Machine: An infinite memory tape divided into discrete cells, a read/write head, an alphabet of symbols, an internal state register, and a lookup table of explicit transition rules.

  • Execution Cycle: The machine reads the active symbol $\rightarrow$ applies the matching rule $\rightarrow$ writes or erases a symbol $\rightarrow$ shifts the tape head left or right $\rightarrow$ transitions to a new state $\rightarrow$ repeats.

  • Church-Turing Thesis: Any function that can be mechanically calculated by an agent following a clear procedure can be computed by a Turing machine.

  • Universal Turing Machine (UTM): A Turing machine capable of reading the structural description of any specialized Turing machine from its tape and simulating its execution, establishing the foundation for general-purpose programmable hardware.

• Computational Functionalism (Putnam):

  • Hilary Putnam proposed that mental states (beliefs, desires, intentions) are functional computational states of a physical system, with reasoning corresponding to computational state transitions.

  • While the physical biological brain is not a literal serial Turing machine equipped with a physical tape and mechanical head, the Turing machine framework provides a formal high-level model of deliberate, serial, rule-governed symbolic reasoning.

Neural Networks, Machine Learning, and Optimization

• Classical vs. Neural Network Paradigm:

  • Classical Paradigm: Relies on explicitly programmed, hand-crafted rules and discrete symbolic manipulations.

  • Neural Network Paradigm: Relies on continuous nodes, adaptive weights, activation functions, and empirical learning from raw data.

  • Neural Learning Loop: $\text{Data Input} \rightarrow \text{Prediction} \rightarrow \text{Calculate Error/Loss} \rightarrow \text{Adjust Weights/Parameters} \rightarrow \text{Repeat}$.

• Architecture of an Artificial Neuron:

  • Mathematical Processing: Multiplies input values by numerical weights, sums the weighted inputs, applies a non-linear activation function, and transmits an output activation value.

  • Weights: Numerical parameters controlling the connection strength and directional influence (excitatory or inhibitory) between nodes.

  • Activation Function: Determines a node's continuous output based on its combined weighted inputs.

• Objective Functions vs. Loss Functions:

  • Objective: The macro-level goal the system is designed to achieve (what success means).

  • Loss: The quantitative mathematical metric measuring the magnitude of error between a prediction and the target answer. Lower loss indicates superior model performance.

• Optimization Algorithms:

  • Gradient Descent: An optimization technique that iteratively updates network parameters in the mathematical direction that reduces loss.

  • Backpropagation: Calculates how much each weight contributed to the total prediction error using the chain rule, enabling targeted parameter updates.

  • Biological Caveat: The success of backpropagation in artificial networks does not prove that human brains learn via backpropagation.

• Machine Learning Fundamentals:

  • Shift in Paradigm: Traditional programming inputs human-crafted rules and data to generate answers. Machine learning inputs data and target answers to learn predictive rules and parameters automatically.

  • The Four Core Components ($\text{D-M-O-O}$):

    1. Data: Examples used for learning. Composed of Features (input variables/attributes) and Labels/Targets (the correct output variable).

       • Example (House Prices): Square footage, bedroom count, and location are features; the final sale price is the label.

    2. Model: The functional architecture taking features as inputs to compute predictions.

    3. Objective/Loss Function: The quantitative error metric measuring performance.

    4. Optimization Algorithm: The parameter adjustment mechanism that iteratively reduces loss.

• Dataset Division and Overfitting:

  • Training Data: Dataset used by the algorithm to adjust internal parameters.

  • Test Data: Held-out dataset used to evaluate the model's ability to generalize to unseen instances.

  • Overfitting: Occurs when a model achieves near-zero error on training data but performs poorly on test data.

    • Analogy: A student who memorizes exact practice exam questions rather than learning underlying concepts.

• Taxonomy of Machine Learning Paradigms:

  • Supervised Learning: Learning from data paired with explicit ground-truth labels.

    • Regression: Predicts continuous numerical targets (e.g., house prices, rainfall amounts, hospital stay lengths). Answers "How much?" or "How many?"

    • Classification: Predicts discrete categorical labels (e.g., spam vs. non-spam, cat vs. dog, disease presence vs. absence). Answers "Which category?"

  • Unsupervised Learning: Discovers hidden structures, patterns, or groupings within data lacking explicit target labels.

  • Reinforcement Learning: An autonomous agent learns an optimal behavioral policy by taking actions in an environment to maximize numerical rewards and minimize penalties.

  • Deep Learning: Uses architectures with many successive transformation layers to learn representation hierarchies. Highly effective for complex data (natural language, computer vision, speech). Depends heavily on large datasets; biased or unrepresentative training data yields biased predictions ("garbage in, garbage out").

• AI Engineering vs. Cognitive Science Modeling:

  • AI Engineering Goal: Build functional systems that execute intelligent tasks, regardless of internal implementation.

  • Cognitive Science Goal: Model and explain the actual computational and neural mechanisms driving human cognition.

  • Helicopter vs. Bird Analogy: Both helicopters and birds achieve flight, but they use entirely different mechanical principles.

  • Weak Modeling: Models that reproduce human behavioral outputs without matching underlying internal cognitive mechanisms.

  • Strong Modeling: Models designed to capture both human behavior and the actual underlying cognitive mechanisms.

  • Core Principle: Demonstrating identical behavioral outputs ($\text{Same Behavior}$) does not establish identical internal mechanisms ($\text{Same Mechanism}$).

Frontier Models, Attention Mechanisms, and World Models

• The Core Frontier Debate: Do deep neural networks (e.g., large language models) acquire internal, structural world models, or do they rely on surface-level statistical associations?

• Three Internal Representations of Domain Rules:

  • World Model: A unified, relational representation capturing the underlying entities, causal structures, and laws governing a domain.

  • Heuristics: Partial, task-specific rules of thumb relying on superficial cues.

  • Memorization: Stored input-output responses without underlying structural representation.

• The Othello-GPT Experiment (Kenneth Li):

  • Setup: A transformer model is trained exclusively on text sequences of valid Othello moves (e.g., $\text{E3} \rightarrow \text{D3} \rightarrow \text{C4}$).

  • Informational Deprivation: The model is never shown the 2D visual board, given explicit game rules, or told piece colors (black vs. white).

  • Task: Predict the next legal move token.

  • Result: Post-training, Othello-GPT predicts legal moves with $99.98\%$ accuracy.

• Embeddings in Othello-GPT:

  • Move Embedding: A $512$--dimensional vector space encoding the played board square (row, column, corner/edge/center position, aligned squares, temporal play tendencies).

  • Position Embedding: A $512$--dimensional vector space encoding sequence timing (move count, active player's turn, game phase [opening, midgame, endgame], board fullness).

  • Vector Combination: The move embedding and position embedding are combined via vector addition: $\text{Move Embedding} + \text{Position Embedding}$.

• Transformer Attention Mechanism:

  • Query Vector ($Q$): "What am I looking for?" (Represents active context and search target).

  • Key Vector ($K$): "Is this relevant?" (Calculates relevance via dot-product matching with queries).

  • Value Vector ($V$): "What information is here?" (Contains the actual vector content extracted to update representations).

• Probing and Causality:

  • Probe Definition: A simple diagnostic classifier trained on a network's internal node activations to decode encoded information.

  • Probing Results: Probes decoded the full 2D Othello board state (tile states: black, white, empty) from Othello-GPT's intermediate activations with high accuracy.

  • Intervention Experiment: Because the presence of decoded information does not prove the model uses it, researchers artificially edited internal representations of board tiles. The model's move predictions changed in exact alignment with the edited board state, proving the internal representation causally drives predictions.

• Limitations and Counterarguments (Melanie Mitchell):

  • Domain Limitations: Othello is a synthetic, closed domain with finite rules. Success in Othello does not establish that real-world LLMs possess human-like world models across complex open domains.

  • Spurious Correlations / Shortcuts: AI models often achieve high performance by exploiting irrelevant shortcuts. For example, a skin-lesion classifier identified malignant tumors by detecting the presence of visual calibration rulers in medical training images.

  • Spectrum of World Models: World models exist on a continuum: $\text{Lookup Table} \rightarrow \text{Map} \rightarrow \text{Dynamic Model} \rightarrow \text{Full Simulator}$.

  • Takeaway: High behavioral accuracy alone does not prove genuine world understanding.

Representational Content, Format, and the Mental Imagery Debate

• Minimal Definition of Representation:

  1. It is about an entity, event, or state of affairs (possesses intentionality).

  2. It possesses truth-evaluable accuracy (can be accurate or inaccurate).

  • Vehicles: Sentences, symbols, photographs, maps, statues, paintings, and diagrams.

• Content vs. Format Distinction:

  • Content: What is represented (the core meaning or semantic referent).

  • Format: How information is represented (the structural vehicle).

  • Illustration: The content "Chris likes cheese" remains identical across different representational formats, such as the English sentence "Chris likes cheese", the French sentence "Chris aime le fromage", ANSI numerical values, or binary Morse code.

• The Mental Imagery Debate:

  • Core Question: What format do internal mental imagery representations use?

  • Zenon Pylyshyn (Propositional View): Mental imagery relies on sentence-like, symbolic descriptions (propositions) that encode spatial relationships abstractly without spatial structure.

  • Stephen Kosslyn (Depictive View): Mental imagery relies on depictive, map-like representations where spatial relationships in the mental image map directly onto functional spatial arrangements in neural activation patterns. Kosslyn emphasizes that depictive representation does not imply a literal physical picture inside the brain, but rather a functional visual buffer preserving spatial layout.

• John R. Anderson's Representational Equivalence Challenge:

  • Behavioral data (such as mental scanning speeds or rotation reaction times) can often be modeled by propositional systems equipped with specific processing rules just as easily as by depictive systems.

  • Because behavioral evidence can be explained by multiple distinct representational systems, behavioral data alone cannot uniquely prove internal representational format.

• Individual Differences: Aphantasia and Hyperphantasia (Zeman):

  • Aphantasia: Total or near-total absence of conscious visual mental imagery, affecting approximately $1\%$ of the population.

  • Hyperphantasia: Condition characterized by exceptionally vivid, lifelike visual mental imagery.

  • Top-Down Generation: Visual mental imagery is a top-down process ("vision in reverse"), generating experiences without direct sensory input.

  • Objective Physiological Evidence: Aphantasia is validated beyond self-reports via distinct binocular rivalry patterns, physiological skin conductance during scary stories, and pupil size adjustments when imagining bright or dark scenes.

  • Correlates: Reduced vividness of autobiographical memories, less vivid visual future imagination, and mild facial recognition difficulties.

  • Crucial Finding: The absence of conscious visual mental imagery does not impede normal high-level cognition, imagination, reasoning, or intelligence.

Theoretical Approaches to Mental Content and Language of Thought

• Philosophical Theories of Representational Content (Ravenscroft):

  • The Intentionality Question: How does a mental state obtain semantic content (become about something)?

  • 1. Resemblance Theory:

    • Thesis: Representation $X$ means $Y$ because $X$ physically or structurally resembles $Y$.

    • Problems: Cannot account for abstract concepts (e.g., inflation, negative numbers, the number $2$). Furthermore, resemblance is symmetric ($X$ resembles $Y$ implies $Y$ resembles $X$), whereas representation is asymmetric.

  • 2. Causal Theory:

    • Thesis: Content is determined by causal chains: representation $X$ means $Y$ because instances of $Y$ cause activations of $X$ (e.g., actual dogs cause DOG representations).

    • Fatal Flaw (The Disjunction Problem): If an observer mistakes a sheep in poor lighting for a dog, the sheep caused a DOG representation. Under Causal Theory, the meaning of DOG becomes the disjunction "dog OR sheep". Causal theory fails to account for misrepresentation or perceptual error.

  • 3. Teleological Theory:

    • Thesis: Representational content is determined by evolutionary biological function—what a representational state was selected by natural selection to detect.

    • Advantage: Explains misrepresentation (an error occurs when a mechanism fails to execute its evolved biological function).

    • Problem (Indeterminacy): Difficult to uniquely identify the exact historical selective pressure that established the representation.

• Language of Thought (LOT) Hypothesis:

  • Compositionality: Natural language is compositional; the semantic meaning of a complex expression depends systematically on the meanings of its primitive tokens and syntactic structure.

  • Mentalese Architecture: The Language of Thought hypothesis asserts that human thought is mediated by an internal, language-like representational system ("Mentalese").

  • Explanatory Power: Provides a mechanistic account for Systematicity (ability to represent $R(a,b)$ entails ability to represent $R(a,c)$) and Productivity (generating infinite thoughts through recursive combination of discrete symbols).

Essential Cognitive Science Terminology and Core Principles

• Master Definitions Table:

*   **Computational:** Goal / What / Why.
*   **Representation & Algorithm:** Information + Format + Rules / How.
*   **Implementation:** Physical Realization.
*   **Systematicity:** Ability to rearrange components to form new representations.
*   **Productivity:** Ability to recursively combine components continuously.
*   **Declarative Memory:** Memory for explicit facts and knowledge.
*   **Procedural Memory:** Memory for rules, skills, and procedures.
*   **Weak Modeling:** Replicating human behavioral outputs without matching internal cognitive mechanisms.
*   **Strong Modeling:** Replicating human behavioral outputs while capturing identical underlying cognitive mechanisms.
*   **Objective:** Definition of what success means in a computational system.
*   **Loss:** Mathematical metric evaluating how wrong a prediction is.
*   **Training Data:** Dataset used by a system to learn parameters.
*   **Test Data:** Dataset used to evaluate generalization to unseen instances.
*   **Overfitting:** Pattern characterized by excellent performance on training data but poor performance on test data.
*   **Regression:** Predictive modeling targeting continuous numerical values.
*   **Classification:** Predictive modeling targeting discrete categories.
*   **Supervised Learning:** Machine learning conducted using explicit input-label pairs.
*   **Unsupervised Learning:** Machine learning conducted on data without pre-existing labels.
*   **Reinforcement Learning:** Machine learning driven by actions, environment interaction, and rewards.
*   **Query ($Q$):** Attention vector specifying "What am I looking for?"
*   **Key ($K$):** Attention vector specifying "Is this relevant?"
*   **Value ($V$):** Attention vector specifying "What information is here?"
*   **World Model:** Unified relational representation of environmental structure and causal laws.
*   **Heuristic:** Partial, context-dependent rule of thumb.
*   **Memorization:** Direct storage of input-output response mappings.
*   **Probe:** Diagnostic classifier testing what information is encoded in network activations.
*   **Content:** What is represented (meaning/referent).
*   **Format:** How information is represented (structural vehicle).
*   **Pylyshyn:** Stance that mental imagery relies on sentence/proposition-like representations.
*   **Kosslyn:** Stance that mental imagery relies on picture/depictive-like representations.
*   **Aphantasia:** Condition characterized by little or no conscious visual mental imagery.
*   **Hyperphantasia:** Condition characterized by extremely vivid visual mental imagery.
*   **Resemblance Theory:** Account defining representational content via physical or structural similarity.
*   **Causal Theory:** Account defining representational content via causal triggers.
*   **Causal Theory's Problem:** The Disjunction Problem (inability to handle misrepresentation).
*   **Teleological Theory:** Account defining representational content via biological/evolutionary function.

• Ten Core Sentences to Memorize:

  1. Different explanatory questions require different levels of abstraction.

  2. The Classical Computational Theory of Mind (CCTM) states that the mind is a computational system using discrete symbolic representations and rule-governed algorithms.

  3. Same behavior does not necessarily mean same internal mechanism.

  4. Machine learning learns behavior from data/experience instead of requiring every rule to be explicitly programmed.

  5. Overfitting = excellent training performance but poor generalization to unseen data.

  6. Regression predicts numbers; classification predicts categories.

  7. Query seeks information, key determines relevance, value contains the information.

  8. High behavioral accuracy alone does not prove a model has a world model.

  9. Content is what a representation represents; format is how it represents it.

  10. The imagery debate concerns whether internal imagery representations are proposition/sentence-like or depictive/picture-like.