Actions and Habits in Instrumental Conditioning
Foundations of Instrumental Behavior and Choice
Determinants of Instrumental Acquisition and Performance:
Response-Reinforcer Contiguity and Contingency: The temporal proximity (contiguity) and the causal relationship (contingency) between the execution of a response and the delivery of a reinforcer fundamentally dictate the speed and strength of behavioral conditioning.
Response-Reinforcer Relevance (Belongingness): The biological predispositions of an organism dictate that certain responses naturally belong with specific classes of reinforcers (e.g., locomotor activities paired with food acquisition, defensive actions paired with shock avoidance).
Reinforcer Magnitude: Quantitative differences in reward magnitude directly scale with performance speed, response rate, and resistance to extinction.
Reinforcer Contrast and Shifts: Behavioral output is dynamically modulated by past reward history; unexpected increases or reductions in reinforcer value generate behavioral contrast effects that alter performance beyond absolute reward values.
Concurrent Schedules and the Modeling of Choice:
In natural environments, organisms rarely encounter singular behavioral options in isolation; time and energy must be distributed across multiple competing alternatives.
In laboratory paradigms, choice behavior is formally investigated using concurrent schedules of reinforcement, where two or more schedules operate simultaneously, each linked to a distinct operant response.
Empirical evaluation demonstrates that subjects systematically allocate their behavioral effort in proportion to the relative reinforcement rates earned from each respective alternative.
The Matching Law:
Definition: Formulated to mathematically quantify choice, the Matching Law states that the relative rate of responding to an option matches the relative rate of reinforcement obtained from that option: where and denote the response frequencies (behaviors) directed toward options and , and and denote the reinforcement frequencies obtained from options and .
Predictive Validity: The Matching Law successfully models and predicts behavioral allocation across diverse species, including animal foraging choices and complex human decision-making scenarios.
Valuation and Cost-Benefit Computation in Instrumental Choice:
Response choice is determined by computing the relative rate and net value of earned outcomes.
Factors Increasing Reinforcer Value:
Reinforcer magnitude (quantitative yield).
Current biological significance and internal motivational state (e.g., hunger, thirst, thermal regulation).
Costs Offsetting Value: The gross subjective value of a reinforcer is discounted by operational costs, including:
Effort: Physical exertion and cognitive load required to execute the instrumental action.
Delay: Temporal delay separating the response from reinforcer delivery (temporal discounting).
Risk: Probability of incurring negative consequences, such as punishment, resource loss, or omitted reward.
Theoretical Frameworks of Instrumental Learning: S-R vs. R-O
Core Mechanistic Inquiries:
What fundamental associations are encoded within the nervous system during instrumental conditioning?
Which neural representations guide and sustain instrumental action?
Is instrumental conditioning driven by blind, automatic habits or by deliberate, cognitive evaluations of prospective outcomes?
Thorndike's Law of Effect and Stimulus-Response () Theory:
Edward Thorndike methodically investigated the acquisition of novel behaviors through animal puzzle-box experiments, establishing that new motor responses and skills emerge via gradual trial-and-error rather than sudden insight.
The Law of Effect: States that responses followed by a "satisfier" (satisfying state of affairs) become more strongly connected to the prevailing stimulus situation, increasing the future probability of that response in the presence of those stimuli. Conversely, responses followed by a "dissatisfier" (annoying state of affairs) undergo weakening of their connection to the situation, decreasing their future probability.
Associative Structure of Theory:
Instrumental learning consists entirely of an association between antecedent stimuli () and the emitted motor response ().
Function of the Reinforcer: The reinforcer () merely serves as an external catalyst or biological "cement" that stamps in or strengthens the direct bond.
Absence of Outcome Representation: Critically, representations of the reinforcer itself (its sensory attributes, qualitative nature, or current value) are not encoded within the learned associative substrate. Once formed, exposure to antecedent stimuli () automatically and reflexively triggers the response ().

Response-Outcome () Theory and Goal-Directed Action:
In direct contrast to Thorndike's peripheral model, theory posits that animals form an explicit internal association between the instrumental response () and the specific consequence or outcome () it produces.
Cognitive Nature: Learning yields an internal representation of the reinforcer's qualitative, sensory, and motivational properties.
Mechanics of Action Selection: When an organism considers executing an instrumental action, it actively calls to mind the representation of the outcome () and its present incentive value, evaluating whether to perform .

Contrasting Habitual Control vs. Goal-Directed Control:
Stimulus-Response () Responding:
Mechanism: Habitual.
Processing: Automatic and reflexive.
Flexibility: Inflexible; insensitive to sudden shifts in outcome value or causal contingency.
Response-Outcome () Responding:
Mechanism: Goal-directed.
Processing: Cognitive, deliberative, and prospective.
Flexibility: Highly flexible; dynamically updates based on changes in outcome value or contingency rules.
Empirical Challenges to Pure S-R Theory
Crespi (1942) Contrast Experiments:
To test whether behavior reflects a simple mechanical stamping-in of strength or an explicit cognitive representation of expected outcome magnitude, Crespi assessed runway running speeds in rats exposed to shifting reward magnitudes.

Experimental Design:
Phase 1 (Acquisition Training):
Positive Contrast Groups: Distinct groups of rats were trained to traverse an alley runway for low-to-moderate rewards: , , or .
Negative Contrast Groups: Distinct groups of rats were trained to traverse an alley runway for moderate-to-high rewards: , , or .
Phase 2 (Shift Phase):
All groups were abruptly shifted to a uniform reward magnitude of .
Behavioral Findings and Theoretical Significance:
Positive Contrast (Elation Effect): Animals shifted from or up to rapidly accelerated their running speeds, surpassing the baseline performance of control rats that had consistently received from the outset.
Negative Contrast (Depression Effect): Animals shifted from or down to experienced a precipitous decline in running speed, performing significantly slower than control rats maintained continuously on .
Implications for Theory: If performance were merely an accumulation of stamped-in habit strength derived from past reinforcement trials, shifting reward down from to would produce a gradual decay, maintaining higher or equal strength compared to animals trained on . The rapid, exaggerated collapse in running speed proves that animals store an internal expectancy and qualitative evaluation of the outcome, directly falsifying pure models.
Dual-Criterion Framework for Goal-Directed Instrumental Action
Belief and Desire in Goal-Directed Action:
Contemporary learning theory conceptualizes goal-directed instrumental action as the psychological integration of two distinct mental constructs:
Belief: Knowledge of the causal relation linking action to outcome ( association; e.g., the belief that driving to Starbucks produces coffee).
Desire: The incentive valuation assigned to the expected outcome (e.g., the subjective desire for coffee based on current taste preferences and fatigue levels).
Formal Operational Criteria (Anthony Dickinson):
To classify an instrumental behavior as truly goal-directed rather than habitual or reflexive, an action must satisfy two formal empirical criteria:
The Contingency Criterion: Performance of the action must depend on an explicit representation of the causal contingency between the response () and the consequence ().
The Goal Value Criterion: Performance of the action must depend on the current biological and motivational incentive status of the outcome ().
Dickinson (1994) Statement: "…goal-directed action represents the most basic behavioral marker of cognition. And the evidence reviewed… suggests that the standard instrumental or operant procedures of the animal conditioning laboratory can establish goal-directed actions, mediated by the interaction of knowledge of the action-outcome relation with knowledge about status of the outcome as a goal." (p. 74).
Testing the Contingency Criterion: Contingency Degradation
Principles of Contingency Degradation:
If an instrumental action is mediated by an belief, degrading the predictive and causal efficacy of the response should selectively extinguish that specific behavior, even if the absolute rate of reward presentation remains constant.
Causal contingency is defined by the difference between two conditional probabilities:
Probability of outcome given the response: , or .
Probability of outcome occurring spontaneously/freely in the absence of the response: , or .
Hammond (1980) Empirical Demonstration:
Methodology: Thirsty rats were trained on a variable interval schedule where lever pressing () produced water delivery.

Experimental Phases:
Phase 1 (Positive Contingency: ): Each second, the probability of water given a lever press was , whereas the probability of non-contingent (free) water was . Under this condition, subjects exhibited vigorous operant responding (~).
Phase 2 (Degraded Contingency: ): The contingency was degraded by delivering free water non-contingently at a rate of , matching the active pressing rate . Because water was delivered with equal probability whether the animal pressed the lever or remained idle, the response-reinforcer contingency was reduced to zero (). Responding immediately collapsed toward zero (~).
Phase 3 (Reinstatement of Contingency: ): Non-contingent water was eliminated (), restoring the causal contingency. Operant responding immediately rebounded (~).
Phase 4 (Re-Degradation: ): Zero-contingency was reinstated, causing response rates to drop back to baseline extinction levels.
Significance: The immediate sensitivity of instrumental responding to the degradation of contingency proves that the behavior is controlled by knowledge of the causal response-outcome relationship (), satisfying the first criterion for goal-directed action.
Testing the Goal Value Criterion: Outcome Devaluation Paradigms
Theoretical Predictions for Outcome Devaluation:
To dissociate from control, the value of the outcome is degraded independently of the instrumental response. The critical test must occur under extinction conditions (where no reinforcers are delivered), ensuring the animal cannot learn a new association during the test itself.
Hypothesis: Because the outcome is not encoded within the associative bond, devaluing the reinforcer outside of the test context should have no immediate effect on response output. The antecedent stimuli () should continue eliciting the motor response () until new non-reinforcement learning takes place.
Hypothesis: Because performance is guided by prospective outcome valuation, an animal recalling the devalued outcome will immediately suppress the corresponding response (), even on the very first extinction trial.
Conditioned Taste Aversion (CTA) Devaluation via Illness (Adams, 1981):
Training Phase: Rats were trained to press a lever for food pellets (.
Devaluation Phase: Outside the operant chamber, the food pellet was devalued in one group using lithium chloride () injections to induce gastric malaise:
Group Devalued (DEV): Paired presentations of Food Pellet with -induced illness (establishing taste aversion).
Group Non-Devalued (NON): Unpaired presentations of Food Pellet and illness (maintaining food pellet palatability).
Extinction Test Phase: Both groups were returned to the operant chambers and allowed to press the lever in complete extinction (no pellets delivered; ).

Empirical Results:
Group DEV pressed the lever at a rate of approximately .
Group NON pressed the lever at a rate of approximately .
Conclusion: The immediate, selective suppression of lever pressing in Group DEV confirms that the action was mediated by an association; the animal integrated its updated negative valuation of the outcome to proactively suppress behavior without requiring experiential punishment at the lever.
Sensory-Specific Satiety Devaluation:
Training Phase: Animals are trained on an instrumental schedule (e.g., .
Devaluation Phase: Prior to testing, animals receive of unlimited pre-feeding access to either:
Group DEV: Chocolate Pellets (satiating the specific sensory attributes of the instrumental reward).
Group NON: Sucrose Pellets (satiating an irrelevant caloric reward, leaving chocolate value intact).
Extinction Test Phase: Subjects are tested on the lever in extinction ().
Results: Group DEV exhibits a profound suppression of lever pressing relative to Group NON, corroborating that instrumental actions are dynamically calibrated to the real-time biological value of specific reward goals.
Real-World Examples of Goal-Directed Action and Devaluation:
Selecting dinner on a mobile food delivery application (e.g., browsing diverse cuisines such as Vietnamese, Thai, Chinese, Sushi, Ramen, Mexican, or Pizza) represents a goal-directed decision mediated by belief ().
If an individual consumes excessive pizza and develops acute digestive distress or postprandial nausea, the incentive value of pizza undergoes devaluation. Future food selection immediately eliminates pizza options in favor of alternative food categories, mirroring the laboratory devaluation effect.
The Transition from Goal-Directed Actions to Autonomous Habits
Characteristics of Goal-Directed vs. Habitual Behaviors:
Attribute | Goal-Directed Action () | Habitual Behavior () |
|---|---|---|
Associative Structure | Response-Outcome () | Stimulus-Response () |
Cognitive Basis | Cognitive, reflective, deliberative | Automatic, reflexive, mechanical |
Temporal Orientation | Prospective evaluation of consequences | Retrospective execution based on past history |
Flexibility | Highly flexible to environmental changes | Inflexible; resistant to rapid alteration |
Devaluation Sensitivity | Sensitive (devaluation suppresses ) | Insensitive (devaluation does not suppress ) |
Resource Utilization | Demands high cognitive/neural bandwidth | Low cognitive demand; resource efficient |
Motor Learning Analogy: The Transition to Autopilot:
When a child initially learns to ride a bicycle, the behavior is strictly goal-directed: each foot movement is executed deliberately based on anticipated outcome representations ("Push right foot downward to balance and propel bike forward; push left foot to maintain momentum").
Following extensive practice, the physical contact and sensory feedback of the pedal acts as an antecedent stimulus () that automatically triggers the mechanical pedaling response (). The skilled cyclist can engage in complex phone conversations because the low-level motor execution has shifted entirely to autonomous control, freeing central cognitive capacity.
Empirical Demonstrations of Habit Formation via Overtraining
Adams (1982) Overtraining Experiment:
To test whether extended training causes an instrumental behavior to transition from an goal-directed action into an autonomous habit, Adams varied the amount of initial training prior to outcome devaluation.
Phase 1 (Instrumental Training):
Group 100: Moderately trained rats received of .
Group 500: Overtrained rats received of .
Phase 2 (Conditioned Taste Aversion Devaluation):
Subgroup Paired (P): Received sucrose paired with -induced illness (devalued).
Subgroup Unpaired (U): Received sucrose and injections separated temporally (control/valued).
Test Phase: All animals were tested in extinction (). Performance was recorded as the ratio of response rate during test relative to baseline response rate.

Empirical Findings:
Group 100 (Moderately Trained): Displayed robust sensitivity to outcome devaluation. The Paired () subgroup responded at a low ratio of baseline (~), while the Unpaired () subgroup responded at a high ratio (~). Responding remained strictly goal-directed ().
Group 500 (Overtrained): Displayed complete insensitivity to outcome devaluation. The Paired () subgroup responded at a high baseline ratio (~), matching or exceeding the Unpaired () subgroup (~). Despite the aversion to sucrose, animals mechanically pressed the lever.
Conclusion: Extended training transitions behavior from a goal-directed association to an autonomous habit that is resistant to changes in outcome value.
Human Habit Formation: Tricomi, Balleine, & O'Doherty (2009):
Tricomi and colleagues evaluated whether human instrumental responding undergoes an identical temporal shift toward habitual responding following extended training.

Experimental Protocol:
Human participants were trained to perform instrumental actions for specific food outcomes across either a 1-day training or a 3-day extended training regimen.
Following training, selective satiety was utilized to devalue one of the outcomes.
Response rates (measured in responses per second) were evaluated in extinction for both valued and devalued outcomes.
Empirical Findings:
Pre-Devaluation Baseline: Both 1-day and 3-day cohorts displayed equivalent baseline responding (~) across outcome assignments.
Post-Devaluation Extinction Test:
1-Day Training Cohort: Demonstrated intact goal-directed control. Response rates to the valued outcome remained elevated (~), whereas responses directed toward the devalued outcome dropped sharply (~).
3-Day Training Cohort: Exhibited a complete loss of devaluation sensitivity. Responses directed toward the devalued outcome (~) remained indistinguishable from responses directed toward the valued outcome (~).
Significance: Demonstrates evolutionary conservation of the transition from goal-directed action to habit across species; moderate training relies on deliberative evaluation, whereas overtraining precipitates automatic responding.
Neurobehavioral Balance and Clinical Implications of Habit Dominance
The Functional Architecture of Dual Systems:
Organisms operate via an equilibrium between two complementary behavioral systems:
The Goal-Directed System (): Maximizes adaptive flexibility, permitting immediate calibration to fluctuating motivational states, novel contingencies, and environmental shifts.
The Habit System (): Maximizes computational speed and energetic efficiency, offloading routine, stereotypic motor repertoires from conscious executive control.
Disruption of the Goal-Directed vs. Habit Balance:
Under normal physiological states, the goal-directed system can override habitual tendencies when situational demands require deliberate intervention.
Pathological or environmental insults can disrupt this balance, resulting in an overreliance on habit circuits and an inability to assert cognitive control:
Chronic Stress: Prolonged exposure to psychological or physical stressors accelerates habit formation, rendering behaviors autonomous and resistant to cognitive devaluation even after brief training periods.
Addictive Substances: Chronic self-administration of drugs of abuse causes neuroadaptations that degrade cortical goal-directed networks, biasing behavioral control toward compulsive, insensate motor habits.
Comprehensive Synthesis of Associative Paradigms:
Pavlovian Conditioning: Organisms learn predictive environmental relationships (; / ) to execute preparatory, anticipatory responses to unconditioned stimuli.
Instrumental Conditioning: Organisms learn operant behaviors to exert control over their surroundings ( and ).
Early in learning, behavior is goal-directed, satisfying both the contingency criterion (verified via contingency degradation) and the goal value criterion (verified via outcome devaluation).
With repeated overtraining, behavior transitions into an autonomous habit, triggered directly by antecedent contextual stimuli and executed independently of outcome value.