Conditioning and Learning Final

Parts of brain involved in procedural memory

Dorsal Striatum, Globus pallidus (external and internal segments), and Subthalamic nucleus.

Skill

The ability to perform a task. Also referred to as procedural memory. It refers to the ability to follow a “procedure” or a sequence of meaningful steps in the pursuit of a goal.

Sensory-motor skills

Learned movement patterns that are controlled and guided by sensory inputs. Examples include how to hit a curve ball, how to spike a volleyball, how to play an instrument.

Cognitive procedural skills

Some aspects of cognitive skills that require following a sequence of steps or rules. Examples include reasoning and language (specifically grammar).

Fitts and Posner’s 3 stage model of procedural memory acquisition

Cognitive stage, associative stage, autonomous stage

Cognitive stage

Stage one of Fitts and Posner’s 3 stage model. It involves development and practice of a movement pattern. Actions are consciously selected to achieve a goal. Movements in this stage are clumsy and slow, need explicit conscious control, and are sometimes even rehearsed verbally.

Associative stage

Stage two of Fitts and Posner’s 3 stage model. This stage involved a refinement of the movement pattern. Many stereotype movements are learned during this stage but conscious control is still exerted to determine the correct sequence and monitor performance.

Autonomous stage

Stage three of Fitts and Posner’s 3 stage model. This stage involves performance of a movement pattern being essentially automatic. It includes any regular behavior that requires little or no thought that is learned rather than innate.

Procedural skills studies in rodents

Shown that skill training produces distinct patterns of neural activity in the basal ganglia in early (cognitive) and late (autonomous) stages of learning.

Procedural skills studies in humans

Shown that procedural skills, like episodic memories, can undergo sleep-dependent memory consolidation, strengthening skills during sleep.

Transfer of procedural memory skills

There is less transfer regarding skills than there is regarding episodic memories. This means effects of training are specific to the training task although some transfer does occur.

Two levels of anatomical organization within the dorsal striatum (within basal ganglia)

Matrix-Striosomes subdivisions and Dorsolateral-dorsomedial subdivisions.

Matrix subdivision of the dorsal striatum

Larger, more evenly distributed compartment, involved in processing sensory-motor and cognitive information and in procedural learning.

Cells in the striosomes of the dorsal striatum

Have different types of neurotransmitter receptors, connectivity, and gene expression. The function of this component is unknown but it has been found to be selectively vulnerable in neurological and neuropsychiatric disorders such as Huntington disease and drug addiction.

Dorsolateral subdivision

Involved in late/autonomous phase of skill acquisition/habits.

Dorsomedial subdivision

Involved in early/cognitive stage of skill acquisition.

Dorsolateral and dorsomedial subdivisions

Form ‘long-range’ circuits with other brain regions. These circuits are direct and indirect loops from the neocortex and back.

Direct/Excitatory/’Go’ pathway

Neocortex -> Dorsal Striatum -> Globus Pallidus (external segment) -> Subthalamic nucleus -> Globus Pallidus (internal segment)

Operant versus classical conditioning

Operant conditioning involves consequences of behavior shaping future behavior while classical conditioning involves pairing a neutral stimulus with an unconditioned stimulus.

Schedules of reinforcement learning

fixed interval, variable interval, fixed ratio, and variable ratio

Fixed interval

Reinforcement after a fixed amount of time. Responses increase as the time for reinforcement approaches, leading to a "scalloped" pattern. Tend to generate low response rates, as individuals might engage in the behavior primarily just before reinforcement. Also, the behavior tends to diminish quickly if reinforcement stops.

Scalloping effect

FI schedules often lead to this kind of pattern, where response rates are lower right after reinforcement but increase as the interval approaches its end.

Variable interval

Reinforcement after varying amounts of time. This schedule produces more consistent responses as the time for reinforcement is unpredictable. Tend to lead to a moderate (not high) rate of behavior, as individuals are aware that checking too frequently doesn’t increase the likelihood of reward.

Fixed ratio

Reinforcement after a specific number of responses (e.g., a reward after every 10 tasks). This schedule produces high, consistent response rates. Based on achieving a certain number of responses; they are goal-oriented rather than based on the passage of time. It’s important to transition to more variable schedules over time to maintain the behavior and prevent dependency on rewards. This can help sustain the desired behavior even when rewards are not always present.

Variable ratio

Reinforcement after an unpredictable number of responses. This schedule leads to very high and resilient response rates due to the uncertainty of reward.

Shaping

A gradual process of reinforcement in which behaviors that are progressively closer to the target behavior are reinforced, enabling the learner to acquire complex behaviors through incremental steps.

Shaping strategies for reinforcement

Involves reinforcing small steps toward a desired behavior rather than expecting the final behavior immediately. There’s a risk of accidentally reinforcing behaviors that are not part of the desired outcome. For instance, if reinforcement is provided inconsistently or too broadly, the subject might learn behaviors that don’t actually bring them closer to the target behavior. The trainer needs to accurately identify and reinforce each successive approximation to the target behavior while avoiding reinforcing behaviors that don’t contribute to progress.

Ventral striatum

Within the Basal Ganglia. Plays a role in linking responses with outcomes. The main component (Accumbens Nucleus) is highly interconnected with the VTA.

Accumbens Nucleus

Studies led to the idea that this is the ‘pleasure center of the brain’. Stimulation of certain subsections leads to increase or avoidance of behaviors. These properties are largely due to dopaminergic projections received from the VTA.

Ventral tegmental area

Within the midbrain. Important for reinforcement and punishment. Central to reward signaling and operant conditioning.

Medial forebrain bundle

Includes axons that project from VTA to nucleus accumbens (ventral striatum) and other brain regions involved in operant conditioning.

Addiction

Involves the “hijacking” of the reward system by drugs of abuse that leads to the learning of conditioned responses to drug-associated stimuli.

Atkinson-Shiffrin Memory Model

Proposes 3 systems functions that preserve memory at different time scales. (Sensory memory, short-term memory, long-term memory)

Sensory memory

Brief, transient sensations of what you have just perceived. There is one for each sense (neural basis: sensory neocortex). Extremely rapid decay (~1s) and rapidly overwritten by new incoming sensory information. Can transfer to short-term memory (with attention, rehearsal)

Working memory

A transient form of memory that allows for the manipulation of stored information. Refers to information we are currently working with. It is important for complex cognitive tasks such as reasoning problem, and decision making

Working memory properties

Multisensory, small capacity, quick decay (<1 min), easily disrupted by distractions, attention and rehearsal can help preserve but items are easily displaced by new info or distractions, can transfer to/from long-term memory storage.

Long-term memory properties

Not currently in conscious, access to information is slower, larger capacity, info forgotten more slowly.

Working memory capacity measure

“Digit span” test which involves reading a list of numbers that increases in length until an error is made.

Why was Alan Baddeley's Model of Working Memory developed?

To address limitations of early models, specifically those that treated short-term memory as a single, unified storage system.

Alan Baddeley's Model of Working Memory

Involves three components: the phonological loop (inner voice), the sketchpad (inner eye), and the central executive (internal attention).

Tests to assess working memory

Repeating numbers mentally requires a phonological loop and diagraming numbers mentally requires a visuospatial sketchpad.

Tests to assess executive function

Any test that requires manipulation of working memory buffers. Tests exertion of control over behaviors and ability to provide complex organization in response to environmental demands.

Frontal Cortex

Largest of the major cortical lobes of the brain in mammals.

Prefrontal cortex

The most anterior sector of the frontal cortex and plays a prominent role in working memory and executive functioning.

Prefrontal cortex anatomical subdivisions

Orbital, Medial, and Lateral

Lateral prefrontal cortex subdivisions

Dorsolateral and ventrolateral

Dorsal Prefrontal Cortex

Controls maintenance and manipulation of information

Ventral Prefrontal Cortex

Controls maintenance of information

Left Ventrolateral Prefrontal Cortex

Responsible for phonological loop and in coordination with language areas.

Language areas within the left ventrolateral PFC/phonological loop

Broca’s area and Wernicke’s area

Broca’s area

Region in the frontal lobe of the dominant hemisphere, usually the left, of the brain with functions linked to speech production

Wernicke’s area

Region in the temporal lobe of the dominant hemisphere. Controls the ability to understand language.

Right ventrolateral Prefrontal Cortex

Responsible for visuospatial sketchpad in coordination with visual areas.

Visual areas in the visuospatial sketchpad/right ventrolateral PFC

Visual cortex in Occipital Lobe

How does the PFC accomplish working memory?

Sustained neuronal activation and short-term plasticity in PFC neurons contribute to holding those representations in working memory. Activity in PFC neurons represents abstract categories and rules. Interacts with other cortical regions: PFC helps maintain activity in language and sensory areas of the cortex to temporarily represent memories

Delay neurons

Neurons in PFC that present persistent spiking activity during retention of information in working memory

Neural basis of neocortical schemas

Nearly one-third of neurons  in prefrontal cortex were shown to be category-selective and responded specifically to one category but not the other in a categorization experiment. Some can encode numbers, mathematical rules, abstract rules, and rule shifts.

Phineas Gage

Railroad construction worker who survived a severe workplace accident that drove an iron rod through his left frontal cortex. Gage survived the injury, becoming one of the earliest documented cases of brain trauma. Friends and co-workers observed drastic personality shifts, including increased impulsivity, lack of restraint, and difficulty with decision-making

Contribution of Gage case

the notion that specific brain regions are responsible for particular mental functions, a concept that shaped early psychiatric approaches.

Thorazine

The first antipsychotic medication introduced for clinical use in the early 1950s. Originally developed as an anesthetic, it was found to have a calming effect on patients and was soon used to treat symptoms of psychosis.

Executive control

Refers to the mental processes that help us plan, focus, make decisions, and adapt to new or unexpected situations. It allows us to manage our thoughts, behaviors, and emotions in a goal-directed way, especially when tasks are complex.

Abstract representations and working memory

These give the PFC a crucial role in executive control. By relying on these two properties, the PFC serves as a mental workspace where we can hold and manipulate information in real-time, including the symbolic manipulation of information (i.e., use of mathematical rules).

Planning

The PFC is crucial for setting goals, predicting outcomes, organizing steps, and adjusting plans as needed, all of which enable effective, goal-directed actions. In controlling these functions, neurons in PFC interact with neurons in the basal ganglia, thalamus and VTA. The PFC interacts with the dorsomedial striatum in planning actions that are under strong cognitive control (early stages of skill learning). It also interacts with ventral striatum and VTA to anticipate potential rewards/punishments to determine which goals to choose.

PFC planning’s importance to behavioral economics

Pioneered by Daniel Kahneman and Amos Tversky in the 1970s. Their groundbreaking research on cognitive biases revealed that people often deviate from rational decision-making in predictable ways. Several of their experiments demonstrated how strongly memory influences our decisions.

Availability heuristic

Refers to the influence of recent or vivid memories on how we make decisions: we tend to make judgments based on how easily examples come to mind.

System 1 of the Two System Thinking Model

Fast, automatic thinking, prone to error:  Often relies on conditioned behaviors and past habits. It involves brain areas like the amygdala and basal ganglia.

System 2 of the Two System Thinking model

Slow, deliberate thinking that can override system 1 and is more reliable but requires more effort. The PFC is crucial for engaging System 2.

Internal (top-down) Attention

Consciously directed and goal-driven, allowing us to focus on specific information based on our intentions or current tasks, regardless of external stimuli. The PFC is key to maintaining this kind of attention by prioritizing goal-relevant information and suppressing distractions. Believed to occur through feedback projections to other cortical regions.

Cocktail party effect

Example of internal/top-down attention, an individual focuses on a particular conversation amidst background distractions, which requires actively filtering out irrelevant sensory information to stay engaged with the chosen speaker.

Bottom-Up Attention

Driven by external stimuli—like a loud noise or bright light—that automatically capture our focus. Doesn’t require deliberate control and is more reflexive. Primarily supported by forebrain and brainstem cholinergic centers.

Response inhibition (impulse control)

The ability to suppress an automatic or planned response that is inappropriate in a given context, and/or actions that interfere with goal-directed behavior

Go/no-go task (GNGT)

A cognitive test that measures a person's ability to inhibit a response. Participants are instructed to respond as fast and accurately as possible to a go stimulus (in this case a C) via button press, while they should not respond to this same stimulus in the rare cases when it was followed by a visual stop signal (in this case an M). Frontal brain regions are mainly activated during successful action restraint (red).

Stroop task

An impulse control test in which participants name the color of the ink used to print a set of words.

Problem-solving

The ability to analyze situations and select the best course of action to achieve a goal. Even simple fact retrieval, such as remembering that ‘the sine function is periodical’ or that ‘London buses are red’, activates dissociated areas for math versus non-math knowledge. Frontal cortex has been shown to be primarily active during intense and prolonged mathematical reflection

Behavioral flexibility

The capacity to respond rapidly to unanticipated changes in the environment. 

Process of behavior flexibility

Individuals must first identify in real time how their surroundings have changed: this requires attention circuits (bottom-up and top-down) and salience detection. If a previous behavior is not appropriate in the new environment, individuals must inhibit previous responses and reconfigure a new strategy.

Wisconsin Card Sorting Test

A cognitive test frequently used to test behavioral flexibility. It tests rule-based flexibility and abstract reasoning. Subjects learn a rule governing the sorting of cards (e.g., by color). After they catch on, however, the rule is changed (e.g., by shape), requiring an updating of working memory and behavior.

Two-Armed or Four-Armed Bandit Task

Primarily tests reward-based learning and adaptability (instead of cognitive flexibility and reasoning like the WCST). It is considered a better test to assess exploration vs. exploitation behaviors. These refer to the balance between trying new options to discover potentially better outcomes (exploration) and using known choices to secure immediate rewards (exploitation), essential for adaptive decision-making and learning.

Wisconsin Card Sorting vs. Two/Four-armed bandit task

The Wisconsin Card Sorting Test is more about rule discovery and flexibility in using rules, rather than a strategic trade-off between exploring and exploiting different options.

Dysexecutive Syndrome (Executive Dysfunction)

Impairments in the brain's executive functions, typically due to damage in the prefrontal cortex.

Symptoms of executive dysfunction

Difficulties with planning: initiating, organizing and carrying out activities, rigidity in thoughts and actions (abnormal perseverance), or impulsivity, poor problem solving, and mood disturbances due to disrupted impulse control: Difficulty in controlling emotions; rapid mood changes may occur. 

Neuropsychiatric disorders associated with executive dysfunction

Attention-Deficit/Hyperactivity Disorder (ADHD), Schizophrenia, and Obsessive-Compulsive Disorder (OCD)

Role of VTA in rewards

Signals the difference between expected and actual rewards. When a behavior or cue leads to a surprising beneficial outcome, this brain region’s response (dopamine release) is strong which facilitates learning the behavior that led to it: through the projections from this brain region to the accumbens (ventral striatum) and PFC. The next time the same situation arises, the remembered value biases internal attention (PFC) towards repeating the behavior (striatum).

Salience

Which stimuli stand out in the environment.

Sensory salience

The quality of a sensory stimulus that makes it stand out from its surroundings and capture our attention. It's a key factor in how we select what to focus on; often investigated in a visual context, but it can also be applied to other senses.

Saliency map

Highlights areas in a scene that contain salient low-level features such as color changes, contrast, edges, etc.

Super colliculus

A structure in the brainstem involved in detecting salient stimuli. The neurons in this region pool inputs from multiple visual system neurons and form a saliency map, which is then relayed to other brain areas.

Role of Novelty in value assignment

Detecting unexpected information is critical for learning, and the brain has developed mechanisms to detect if something is unexpected and ‘tag’ that information as important to be stored in memory.

Key brain regions for novelty-related behavior

Hippocampal regions (HPC), brainstem regions (VTA, raphe), and PFC

Raphe nuclei role in value assignment

Crucial for curiosity and motivated behavior to increase understanding which comes from encountering novel stimuli.

Novelty detection test

Involves presenting sequences of stimuli, where the majority are repeated (standards) and a small fraction are novel (deviant).

Mismatch negativity (MMN) response

An automatic brain response to an unexpected sensory stimulus. It is increasingly important in cognitive and clinical brain research to understand the basic mechanisms of novelty identification and as a biomarker in several diseases.

Experiments used to test MMN response

oddball paradigms or tasks; measured by subtracting the average neural response to standard stimuli from the average neural response to deviant stimuli. This difference reflects the brain's automatic detection of a deviation from the expected sequence.

Predictive processing

A subfield of study in neuroscience that proposes that the brain continuously generates predictions about incoming sensory information, compares those predictions with the sensory input, and updates its internal models or beliefs based on discrepancies (prediction errors).

Prediction errors

Widely regarded as the basis of Mismatch Negativity (MMN) signals in the brain. These are based on prior experiences and stored knowledge, often represented in the form of schemas (mental models) stored in the neocortex.

Perception

Not a passive reception of sensory data; rather, it is an active and dynamic process where the brain continuously attempts to reconstruct the external world based on incoming sensory information. 

Unconscious inference

The idea that the human brain fills in visual information to make sense of a scene. 

Core of predictive processing theories

The idea that the brain develops a model (generative/internal model) of the world that it uses to predict sensory input; that what we perceive is not the true sensory signal, but a rational reconstruction of what the signal should be.

Predictive processing theory implication

The neocortex is organized in a multi-level hierarchy, where each level can identify unexpected information because it is trying to predict the information that is going to receive from lower levels.

Feedback projections

Carry predictions from higher cortical areas to be compared with incoming sensory input at earlier stages of the cortical hierarchy.