Actions and Habits in Instrumental Conditioning

Foundations of Instrumental Behavior and Choice

  • Determinants of Instrumental Acquisition and Performance:

    • Response-Reinforcer Contiguity and Contingency: The temporal proximity (contiguity) and the causal relationship (contingency) between the execution of a response and the delivery of a reinforcer fundamentally dictate the speed and strength of behavioral conditioning.

    • Response-Reinforcer Relevance (Belongingness): The biological predispositions of an organism dictate that certain responses naturally belong with specific classes of reinforcers (e.g., locomotor activities paired with food acquisition, defensive actions paired with shock avoidance).

    • Reinforcer Magnitude: Quantitative differences in reward magnitude directly scale with performance speed, response rate, and resistance to extinction.

    • Reinforcer Contrast and Shifts: Behavioral output is dynamically modulated by past reward history; unexpected increases or reductions in reinforcer value generate behavioral contrast effects that alter performance beyond absolute reward values.

  • Concurrent Schedules and the Modeling of Choice:

    • In natural environments, organisms rarely encounter singular behavioral options in isolation; time and energy must be distributed across multiple competing alternatives.

    • In laboratory paradigms, choice behavior is formally investigated using concurrent schedules of reinforcement, where two or more schedules operate simultaneously, each linked to a distinct operant response.

    • Empirical evaluation demonstrates that subjects systematically allocate their behavioral effort in proportion to the relative reinforcement rates earned from each respective alternative.

  • The Matching Law:

    • Definition: Formulated to mathematically quantify choice, the Matching Law states that the relative rate of responding to an option matches the relative rate of reinforcement obtained from that option:     B1B1+B2=R1R1+R2\frac{B_1}{B_1 + B_2} = \frac{R_1}{R_1 + R_2}     where B1B_1 and B2B_2 denote the response frequencies (behaviors) directed toward options 11 and 22, and R1R_1 and R2R_2 denote the reinforcement frequencies obtained from options 11 and 22.

    • Predictive Validity: The Matching Law successfully models and predicts behavioral allocation across diverse species, including animal foraging choices and complex human decision-making scenarios.

  • Valuation and Cost-Benefit Computation in Instrumental Choice:

    • Response choice is determined by computing the relative rate and net value of earned outcomes.

    • Factors Increasing Reinforcer Value:

    • Reinforcer magnitude (quantitative yield).

    • Current biological significance and internal motivational state (e.g., hunger, thirst, thermal regulation).

    • Costs Offsetting Value: The gross subjective value of a reinforcer is discounted by operational costs, including:

    • Effort: Physical exertion and cognitive load required to execute the instrumental action.

    • Delay: Temporal delay separating the response from reinforcer delivery (temporal discounting).

    • Risk: Probability of incurring negative consequences, such as punishment, resource loss, or omitted reward.

Theoretical Frameworks of Instrumental Learning: S-R vs. R-O

  • Core Mechanistic Inquiries:

    • What fundamental associations are encoded within the nervous system during instrumental conditioning?

    • Which neural representations guide and sustain instrumental action?

    • Is instrumental conditioning driven by blind, automatic habits or by deliberate, cognitive evaluations of prospective outcomes?

  • Thorndike's Law of Effect and Stimulus-Response (S−RS-R) Theory:

    • Edward Thorndike methodically investigated the acquisition of novel behaviors through animal puzzle-box experiments, establishing that new motor responses and skills emerge via gradual trial-and-error rather than sudden insight.

    • The Law of Effect: States that responses followed by a "satisfier" (satisfying state of affairs) become more strongly connected to the prevailing stimulus situation, increasing the future probability of that response in the presence of those stimuli. Conversely, responses followed by a "dissatisfier" (annoying state of affairs) undergo weakening of their connection to the situation, decreasing their future probability.

    • Associative Structure of S−RS-R Theory:

    • Instrumental learning consists entirely of an association between antecedent stimuli (SS) and the emitted motor response (RR).

    • Function of the Reinforcer: The reinforcer (OO) merely serves as an external catalyst or biological "cement" that stamps in or strengthens the direct S−RS-R bond.

    • Absence of Outcome Representation: Critically, representations of the reinforcer itself (its sensory attributes, qualitative nature, or current value) are not encoded within the learned associative substrate. Once formed, exposure to antecedent stimuli (SS) automatically and reflexively triggers the response (RR).


S-R association stamped in by reinforcer
  • Response-Outcome (R−OR-O) Theory and Goal-Directed Action:

    • In direct contrast to Thorndike's peripheral model, R−OR-O theory posits that animals form an explicit internal association between the instrumental response (RR) and the specific consequence or outcome (OO) it produces.

    • Cognitive Nature: Learning yields an internal representation of the reinforcer's qualitative, sensory, and motivational properties.

    • Mechanics of Action Selection: When an organism considers executing an instrumental action, it actively calls to mind the representation of the outcome (OO) and its present incentive value, evaluating whether to perform RR.


R-O association representing outcome in the brain
  • Contrasting S−RS-R Habitual Control vs. R−OR-O Goal-Directed Control:

    • Stimulus-Response (S−RS-R) Responding:

    • Mechanism: Habitual.

    • Processing: Automatic and reflexive.

    • Flexibility: Inflexible; insensitive to sudden shifts in outcome value or causal contingency.

    • Response-Outcome (R−OR-O) Responding:

    • Mechanism: Goal-directed.

    • Processing: Cognitive, deliberative, and prospective.

    • Flexibility: Highly flexible; dynamically updates based on changes in outcome value or contingency rules.

Empirical Challenges to Pure S-R Theory

  • Crespi (1942) Contrast Experiments:

    • To test whether behavior reflects a simple mechanical stamping-in of S−RS-R strength or an explicit cognitive representation of expected outcome magnitude, Crespi assessed runway running speeds in rats exposed to shifting reward magnitudes.


Crespi 1942 runway speed contrast effects
  • Experimental Design:

    • Phase 1 (Acquisition Training):

    • Positive Contrast Groups: Distinct groups of rats were trained to traverse an alley runway for low-to-moderate rewards: 11, 44, or 16 pellets16\,\text{pellets}.

    • Negative Contrast Groups: Distinct groups of rats were trained to traverse an alley runway for moderate-to-high rewards: 1616, 6464, or 256 pellets256\,\text{pellets}.

    • Phase 2 (Shift Phase):

    • All groups were abruptly shifted to a uniform reward magnitude of 16 pellets16\,\text{pellets}.

  • Behavioral Findings and Theoretical Significance:

    • Positive Contrast (Elation Effect): Animals shifted from 11 or 4 pellets4\,\text{pellets} up to 16 pellets16\,\text{pellets} rapidly accelerated their running speeds, surpassing the baseline performance of control rats that had consistently received 16 pellets16\,\text{pellets} from the outset.

    • Negative Contrast (Depression Effect): Animals shifted from 6464 or 256 pellets256\,\text{pellets} down to 16 pellets16\,\text{pellets} experienced a precipitous decline in running speed, performing significantly slower than control rats maintained continuously on 16 pellets16\,\text{pellets}.

    • Implications for S−RS-R Theory: If performance were merely an accumulation of stamped-in S−RS-R habit strength derived from past reinforcement trials, shifting reward down from 256256 to 1616 would produce a gradual decay, maintaining higher or equal strength compared to animals trained on 16 pellets16\,\text{pellets}. The rapid, exaggerated collapse in running speed proves that animals store an internal expectancy and qualitative evaluation of the outcome, directly falsifying pure S−RS-R models.

Dual-Criterion Framework for Goal-Directed Instrumental Action

  • Belief and Desire in Goal-Directed Action:

    • Contemporary learning theory conceptualizes goal-directed instrumental action as the psychological integration of two distinct mental constructs:

    • Belief: Knowledge of the causal relation linking action to outcome (R−OR-O association; e.g., the belief that driving to Starbucks produces coffee).

    • Desire: The incentive valuation assigned to the expected outcome (e.g., the subjective desire for coffee based on current taste preferences and fatigue levels).

  • Formal Operational Criteria (Anthony Dickinson):

    • To classify an instrumental behavior as truly goal-directed rather than habitual or reflexive, an action must satisfy two formal empirical criteria:

    1. The Contingency Criterion: Performance of the action must depend on an explicit representation of the causal contingency between the response (RR) and the consequence (OO).

    2. The Goal Value Criterion: Performance of the action must depend on the current biological and motivational incentive status of the outcome (OO).

    • Dickinson (1994) Statement: "…goal-directed action represents the most basic behavioral marker of cognition. And the evidence reviewed… suggests that the standard instrumental or operant procedures of the animal conditioning laboratory can establish goal-directed actions, mediated by the interaction of knowledge of the action-outcome relation with knowledge about status of the outcome as a goal." (p. 74).

Testing the Contingency Criterion: Contingency Degradation

  • Principles of Contingency Degradation:

    • If an instrumental action is mediated by an R−OR-O belief, degrading the predictive and causal efficacy of the response should selectively extinguish that specific behavior, even if the absolute rate of reward presentation remains constant.

    • Causal contingency is defined by the difference between two conditional probabilities:

    • Probability of outcome given the response: p(O∣R)p(O|R), or p(Rwd∣LP)p(\text{Rwd}|\text{LP}).

    • Probability of outcome occurring spontaneously/freely in the absence of the response: p(O∣∼R)p(O|\sim R), or p(Rwd∣Free)p(\text{Rwd}|\text{Free}).

  • Hammond (1980) Empirical Demonstration:

    • Methodology: Thirsty rats were trained on a variable interval schedule where lever pressing (LPLP) produced water delivery.


Hammond 1980 contingency degradation response curves
  • Experimental Phases:

    • Phase 1 (Positive Contingency: 0.05−0.00.05 - 0.0): Each second, the probability of water given a lever press was p(Rwd∣LP)=0.05p(\text{Rwd}|\text{LP}) = 0.05, whereas the probability of non-contingent (free) water was p(Rwd∣Free)=0.0p(\text{Rwd}|\text{Free}) = 0.0. Under this condition, subjects exhibited vigorous operant responding (~3000−4000 responses/hour3000-4000\,\text{responses/hour}).

    • Phase 2 (Degraded Contingency: 0.05−0.050.05 - 0.05): The contingency was degraded by delivering free water non-contingently at a rate of p(Rwd∣Free)=0.05p(\text{Rwd}|\text{Free}) = 0.05, matching the active pressing rate p(Rwd∣LP)=0.05p(\text{Rwd}|\text{LP}) = 0.05. Because water was delivered with equal probability whether the animal pressed the lever or remained idle, the response-reinforcer contingency was reduced to zero (Δp=0\Delta p = 0). Responding immediately collapsed toward zero (~100−200 responses/hour100-200\,\text{responses/hour}).

    • Phase 3 (Reinstatement of Contingency: 0.05−0.00.05 - 0.0): Non-contingent water was eliminated (p(Rwd∣Free)=0.0p(\text{Rwd}|\text{Free}) = 0.0), restoring the causal contingency. Operant responding immediately rebounded (~2000−3500 responses/hour2000-3500\,\text{responses/hour}).

    • Phase 4 (Re-Degradation: 0.05−0.050.05 - 0.05): Zero-contingency was reinstated, causing response rates to drop back to baseline extinction levels.

  • Significance: The immediate sensitivity of instrumental responding to the degradation of contingency proves that the behavior is controlled by knowledge of the causal response-outcome relationship (R−OR-O), satisfying the first criterion for goal-directed action.

Testing the Goal Value Criterion: Outcome Devaluation Paradigms

  • Theoretical Predictions for Outcome Devaluation:

    • To dissociate S−RS-R from R−OR-O control, the value of the outcome is degraded independently of the instrumental response. The critical test must occur under extinction conditions (where no reinforcers are delivered), ensuring the animal cannot learn a new association during the test itself.

    • S−RS-R Hypothesis: Because the outcome is not encoded within the associative bond, devaluing the reinforcer outside of the test context should have no immediate effect on response output. The antecedent stimuli (SS) should continue eliciting the motor response (RR) until new non-reinforcement learning takes place.

    • R−OR-O Hypothesis: Because performance is guided by prospective outcome valuation, an animal recalling the devalued outcome will immediately suppress the corresponding response (RR), even on the very first extinction trial.

  • Conditioned Taste Aversion (CTA) Devaluation via Illness (Adams, 1981):

    • Training Phase: Rats were trained to press a lever for food pellets (Lever Press→Food Pellet\text{Lever Press} \rightarrow \text{Food Pellet}.

    • Devaluation Phase: Outside the operant chamber, the food pellet was devalued in one group using lithium chloride (LiClLiCl) injections to induce gastric malaise:

    • Group Devalued (DEV): Paired presentations of Food Pellet with LiClLiCl-induced illness (establishing taste aversion).

    • Group Non-Devalued (NON): Unpaired presentations of Food Pellet and illness (maintaining food pellet palatability).

    • Extinction Test Phase: Both groups were returned to the operant chambers and allowed to press the lever in complete extinction (no pellets delivered; Lever Press→∅\text{Lever Press} \rightarrow \emptyset).


Adams 1981 outcome devaluation via illness bar graph
  • Empirical Results:

    • Group DEV pressed the lever at a rate of approximately 1.7 presses/min1.7\,\text{presses/min}.

    • Group NON pressed the lever at a rate of approximately 4.2 presses/min4.2\,\text{presses/min}.

  • Conclusion: The immediate, selective suppression of lever pressing in Group DEV confirms that the action was mediated by an R−OR-O association; the animal integrated its updated negative valuation of the outcome to proactively suppress behavior without requiring experiential punishment at the lever.

    • Sensory-Specific Satiety Devaluation:

  • Training Phase: Animals are trained on an instrumental schedule (e.g., Lever Press→Chocolate Pellet\text{Lever Press} \rightarrow \text{Chocolate Pellet}.

  • Devaluation Phase: Prior to testing, animals receive 1 hour1\,\text{hour} of unlimited pre-feeding access to either:

    • Group DEV: Chocolate Pellets (satiating the specific sensory attributes of the instrumental reward).

    • Group NON: Sucrose Pellets (satiating an irrelevant caloric reward, leaving chocolate value intact).

  • Extinction Test Phase: Subjects are tested on the lever in extinction (Lever Press→∅\text{Lever Press} \rightarrow \emptyset).

  • Results: Group DEV exhibits a profound suppression of lever pressing relative to Group NON, corroborating that instrumental actions are dynamically calibrated to the real-time biological value of specific reward goals.

    • Real-World Examples of Goal-Directed Action and Devaluation:

  • Selecting dinner on a mobile food delivery application (e.g., browsing diverse cuisines such as Vietnamese, Thai, Chinese, Sushi, Ramen, Mexican, or Pizza) represents a goal-directed decision mediated by belief (R→OR \rightarrow O).

  • If an individual consumes excessive pizza and develops acute digestive distress or postprandial nausea, the incentive value of pizza undergoes devaluation. Future food selection immediately eliminates pizza options in favor of alternative food categories, mirroring the laboratory devaluation effect.

The Transition from Goal-Directed Actions to Autonomous Habits

  • Characteristics of Goal-Directed vs. Habitual Behaviors:

Attribute

Goal-Directed Action (R−OR-O)

Habitual Behavior (S−RS-R)

Associative Structure

Response-Outcome (R−OR-O)

Stimulus-Response (S−RS-R)

Cognitive Basis

Cognitive, reflective, deliberative

Automatic, reflexive, mechanical

Temporal Orientation

Prospective evaluation of consequences

Retrospective execution based on past history

Flexibility

Highly flexible to environmental changes

Inflexible; resistant to rapid alteration

Devaluation Sensitivity

Sensitive (devaluation suppresses RR)

Insensitive (devaluation does not suppress RR)

Resource Utilization

Demands high cognitive/neural bandwidth

Low cognitive demand; resource efficient

  • Motor Learning Analogy: The Transition to Autopilot:

    • When a child initially learns to ride a bicycle, the behavior is strictly goal-directed: each foot movement is executed deliberately based on anticipated outcome representations ("Push right foot downward to balance and propel bike forward; push left foot to maintain momentum").

    • Following extensive practice, the physical contact and sensory feedback of the pedal acts as an antecedent stimulus (SS) that automatically triggers the mechanical pedaling response (RR). The skilled cyclist can engage in complex phone conversations because the low-level motor execution has shifted entirely to autonomous S−RS-R control, freeing central cognitive capacity.

Empirical Demonstrations of Habit Formation via Overtraining

  • Adams (1982) Overtraining Experiment:

    • To test whether extended training causes an instrumental behavior to transition from an R−OR-O goal-directed action into an S−RS-R autonomous habit, Adams varied the amount of initial training prior to outcome devaluation.

    • Phase 1 (Instrumental Training):

    • Group 100: Moderately trained rats received 100 trials100\,\text{trials} of Lever Press→Sucrose\text{Lever Press} \rightarrow \text{Sucrose}.

    • Group 500: Overtrained rats received 500 trials500\,\text{trials} of Lever Press→Sucrose\text{Lever Press} \rightarrow \text{Sucrose}.

    • Phase 2 (Conditioned Taste Aversion Devaluation):

    • Subgroup Paired (P): Received sucrose paired with LiClLiCl-induced illness (devalued).

    • Subgroup Unpaired (U): Received sucrose and LiClLiCl injections separated temporally (control/valued).

    • Test Phase: All animals were tested in extinction (Lever Press→∅\text{Lever Press} \rightarrow \emptyset). Performance was recorded as the ratio of response rate during test relative to baseline response rate.


Adams 1982 overtraining and outcome devaluation bar graph
  • Empirical Findings:

    • Group 100 (Moderately Trained): Displayed robust sensitivity to outcome devaluation. The Paired (PP) subgroup responded at a low ratio of baseline (~0.250.25), while the Unpaired (UU) subgroup responded at a high ratio (~0.550.55). Responding remained strictly goal-directed (R−OR-O).

    • Group 500 (Overtrained): Displayed complete insensitivity to outcome devaluation. The Paired (PP) subgroup responded at a high baseline ratio (~0.470.47), matching or exceeding the Unpaired (UU) subgroup (~0.320.32). Despite the aversion to sucrose, animals mechanically pressed the lever.

  • Conclusion: Extended training transitions behavior from a goal-directed R−OR-O association to an autonomous S−RS-R habit that is resistant to changes in outcome value.

    • Human Habit Formation: Tricomi, Balleine, & O'Doherty (2009):

  • Tricomi and colleagues evaluated whether human instrumental responding undergoes an identical temporal shift toward habitual responding following extended training.


Tricomi et al. 2009 1-day versus 3-day training devaluation responses
  • Experimental Protocol:

    • Human participants were trained to perform instrumental actions for specific food outcomes across either a 1-day training or a 3-day extended training regimen.

    • Following training, selective satiety was utilized to devalue one of the outcomes.

    • Response rates (measured in responses per second) were evaluated in extinction for both valued and devalued outcomes.

  • Empirical Findings:

    • Pre-Devaluation Baseline: Both 1-day and 3-day cohorts displayed equivalent baseline responding (~2.1−2.8 responses/s2.1 - 2.8\,\text{responses/s}) across outcome assignments.

    • Post-Devaluation Extinction Test:

    • 1-Day Training Cohort: Demonstrated intact goal-directed control. Response rates to the valued outcome remained elevated (~3.1 responses/s3.1\,\text{responses/s}), whereas responses directed toward the devalued outcome dropped sharply (~1.2 responses/s1.2\,\text{responses/s}).

    • 3-Day Training Cohort: Exhibited a complete loss of devaluation sensitivity. Responses directed toward the devalued outcome (~2.4 responses/s2.4\,\text{responses/s}) remained indistinguishable from responses directed toward the valued outcome (~2.6 responses/s2.6\,\text{responses/s}).

  • Significance: Demonstrates evolutionary conservation of the transition from goal-directed action to habit across species; moderate training relies on deliberative R−OR-O evaluation, whereas overtraining precipitates automatic S−RS-R responding.

Neurobehavioral Balance and Clinical Implications of Habit Dominance

  • The Functional Architecture of Dual Systems:

    • Organisms operate via an equilibrium between two complementary behavioral systems:

    • The Goal-Directed System (R−OR-O): Maximizes adaptive flexibility, permitting immediate calibration to fluctuating motivational states, novel contingencies, and environmental shifts.

    • The Habit System (S−RS-R): Maximizes computational speed and energetic efficiency, offloading routine, stereotypic motor repertoires from conscious executive control.

  • Disruption of the Goal-Directed vs. Habit Balance:

    • Under normal physiological states, the goal-directed system can override habitual tendencies when situational demands require deliberate intervention.

    • Pathological or environmental insults can disrupt this balance, resulting in an overreliance on habit circuits and an inability to assert cognitive control:

    • Chronic Stress: Prolonged exposure to psychological or physical stressors accelerates habit formation, rendering behaviors autonomous and resistant to cognitive devaluation even after brief training periods.

    • Addictive Substances: Chronic self-administration of drugs of abuse causes neuroadaptations that degrade cortical R−OR-O goal-directed networks, biasing behavioral control toward compulsive, insensate S−RS-R motor habits.

  • Comprehensive Synthesis of Associative Paradigms:

    • Pavlovian Conditioning: Organisms learn predictive environmental relationships (CS→USCS \rightarrow US; S−S∗S-S^* / S−OS-O) to execute preparatory, anticipatory responses to unconditioned stimuli.

    • Instrumental Conditioning: Organisms learn operant behaviors to exert control over their surroundings (R→OR \rightarrow O and S−RS-R).

    • Early in learning, behavior is goal-directed, satisfying both the contingency criterion (verified via contingency degradation) and the goal value criterion (verified via outcome devaluation).

    • With repeated overtraining, behavior transitions into an autonomous S−RS-R habit, triggered directly by antecedent contextual stimuli and executed independently of outcome value.