Instrumental Conditioning Procedures and Schedules of Reinforcement

Foundations of Instrumental Conditioning

  • Fundamental Definition and Core Principle:

    • Instrumental conditioning is a core learning process through which an organism's behavior is modified and controlled by its consequences.

    • Unlike classical Pavlovian conditioning, where conditioned responses (CRCRs$) are elicited automatically by a preceding conditioned stimulus (CSCS), instrumental behaviors are emitted by the organism and operate directly upon the environment to produce outcomes.

    • This associative learning mechanism dictates decision-making, adaptive behavior, and voluntary actions across human and non-human animal species.

  • Historical Perspectives and Theoretical Frameworks:

    • Edward L. Thorndike:

    • Investigated instrumental learning using systematic experimental designs.

    • Concluded that learning proceeds via gradual trial and error rather than sudden mental insight.

    • Formulated the Law of Effect, positing that instrumental learning is strictly associative: the organism establishes a link between the environmental context or stimulus (SS) and the emitted behavioral response (RR).

    • The nature of this association is bidirectional:

      • Satisfying, appetitive outcomes strengthen the association between the stimulus (SS) and the response (RR), increasing the likelihood that RR will occur when SS is present.

      • Annoying, aversive outcomes weaken the SS-RR association, decreasing subsequent response probability.

    • B. F. Skinner:

    • Thoroughly characterized the mechanisms of operant conditioning, developing precise experimental methodologies to investigate how behavioral contingencies shape motor repertoires.

    • Established a functional taxonomy based on whether an environmental stimulus is presented or removed and whether the overarching response rate increases or decreases.

  • The Four Operant Contingencies:

    • Positive Reinforcement:

    • The addition (presentation) of an appetitive, desirable stimulus contingent upon the emission of a response.

    • Effect: Increases the future frequency or probability of the target behavior.

    • Negative Reinforcement:

    • The removal, termination, or avoidance of an aversive, noxious stimulus contingent upon the emission of a response.

    • Effect: Increases the future frequency or probability of the target behavior.

    • Positive Punishment:

    • The addition (presentation) of an aversive, unpleasant stimulus contingent upon the emission of a response.

    • Effect: Decreases the future frequency or probability of the target behavior.

    • Negative Punishment:

    • The removal or omission of an appetitive, desirable stimulus contingent upon the emission of a response.

    • Effect: Decreases the future frequency or probability of the target behavior.

  • Properties of Reinforcers and Conditioned Reinforcement:

    • A reinforcer is any stimulus, event, or condition following an action that increases the future probability or intensity of that action.

    • Reinforcers vary widely in quality, magnitude, and sensory modality.

    • Conditioned (Secondary) Reinforcement: An initially neutral stimulus that has acquired reinforcing properties through prior Pavlovian pairings with an unconditioned, appetitive reinforcer (USUS). Such a conditioned stimulus (CSCS) can maintain new operant behaviors in the absence of primary biological reinforcers.

Instrumental Conditioning Methodologies: Discrete Trial vs. Free Operant

  • Discrete Trial Procedures:

    • Defining Characteristics:

    • Each trial presents the organism with a discrete choice or problem.

    • The subject is permitted to perform the instrumental response only once per trial.

    • The pace and structure of the experiment are strictly regulated by the experimenter, who determines trial onset, contextual presentation, and trial termination.

    • Common Experimental Apparatuses:

    • Thorndike's Puzzle Box:

      • Enclosed wooden compartments (e.g., Box A, Box I) equipped with internal mechanical tripwires, treadles, or loops connected via strings and pulleys to an exterior latch.

      • An animal (typically a hungry cat) is placed inside, requiring the manipulation of a specific mechanism to open the door, escape, and access food positioned outside.

      • Primary dependent variable: latency to escape across repeated trials.

    • Straight-Alley Runway Maze:

      • A linear corridor connecting a start box to a goal box containing food reinforcement.

      • Primary dependent variables: running speed or velocity (feet/second\text{feet/second}) and latency to traverse the runway.

      • Typical acquisition trajectory: Running speed increases progressively from 0 feet/second0\,\text{feet/second} on trial 11 up to asymptotic speeds of approximately 3.6 feet/second3.6\,\text{feet/second} over 1616 to 2020 trials.

    • T-Maze:

      • A start alley terminating at a perpendicular intersection yielding two alternative pathways (left arm vs. right arm).

      • Primary dependent variable: spatial choice percentage or preference distribution when alternative arms contain differential reinforcer magnitudes, probabilities, or reward types.


Thorndike's puzzle box apparatus designs, showing Box A and Box I


Running speed in feet per second across trials in a straight-alley runway maze


T-maze apparatus layout showing the choice point between two arms
  • Free Operant Procedures:

    • Defining Characteristics:

    • The organism is placed into an experimental environment where it is free to explore, interact, and emit the operant response repeatedly without experimenter intervention.

    • The behavior occurs naturally as ongoing activity; the experimental subject, rather than the experimenter, determines the pacing and timing of responses.

    • Permits continuous measurement of behavioral rate and temporal distribution of responding over extended sessions.

Operant Chamber Architecture and Apparatus Design

  • The Operant Chamber (Skinner Box):

    • Conceived and developed by B. F. Skinner to standardize and automate the empirical assessment of free-operant responding.

    • Can be electronically wired and programmed to automate virtually any instrumental contingency, including discrete-trial paradigms.


Schematic diagram of an operant chamber (Skinner box) with labeled components
  • Structural Components and Mechanical Subsystems:

    • Response Manipulandum:

    • For rodents: A retractable or stationary metal lever mounted on the chamber wall requiring a specific minimum downward force to close an electrical switch.

    • For pigeons: An illuminated plastic key or disk flush with the wall that registers pecking behavior via microswitches.

    • Reinforcer Delivery Subsystem:

    • Automated pellet dispenser or liquid dipper containing food rewards.

    • Delivery tube directing the reinforcer into an internal food cup, food magazine, or hopper accessible to the subject.

    • Discriminative Stimulus Generators:

    • Visual cue lights (signal lights) mounted above or alongside the manipulandum.

    • Acoustic speakers capable of delivering discrete tones, white noise, or auditory clicks.

    • Aversive Stimulus Delivery Hardware:

    • Floor consisting of parallel stainless-steel bars wired to an external shock generator to deliver controlled footshocks during punishment or avoidance protocols.

Acquisition Techniques: Magazine Training and Shaping

  • Magazine Training:

    • A vital preliminary phase conducted prior to instrumental response training.

    • Involves non-contingent ("free") delivery of reinforcers directly into the food magazine or cup.

    • Pavlovian Conditioning Mechanism:

    • The mechanical sound of the dispenser and the local context of the food cup serve as conditioned stimuli (CSCS) that reliably precede food availability (USUS).

    • Establishes a conditioned approach response, ensuring the subject immediately navigates to and consumes food when the dispenser fires, eliminating acquisition delays during subsequent operant training.

  • Shaping (Reinforcement of Successive Approximations):

    • Operant responses such as lever-pressing or key-pecking have an initial baseline probability of near zero.

    • Shaping is the systematic reinforcement of successive behavioral approximations toward a predetermined terminal target response, while placing earlier approximations on extinction.

    • Standard Procedure:

    1. Identify Starting Point: Observe the organism's baseline behavioral repertoire and identify an existing action remotely oriented toward the target (e.g., facing the wall with the lever).

    2. Identify Terminal Goal: Explicitly define the completed operant response (e.g., downward depression of the lever with a front paw).

    3. Divide the Difference into Sequential Steps: Systematically shift reinforcement criteria:

      • Step 1: Deliver reinforcement when the animal turns toward the lever.

      • Step 2: Reinforce only when the animal moves into physical proximity to the lever.

      • Step 3: Reinforce only when the animal rears up near the lever.

      • Step 4: Reinforce only when the animal's paw makes physical contact with the lever.

      • Step 5: Reinforce exclusively when the lever is depressed with sufficient force to activate the microswitch.

    • Generality and Real-World Applications:

    • Shaping governs the acquisition of complex motor behaviors across humans and other animals.

    • Examples include language acquisition in human infants (progressing from babbling, to word-like utterances, to complete grammatical phrases), athletic technique mastery (such as mastering a complex gymnastic floor pass or a basketball shooting form), and teaching children motor coordination skills.

Behavioral Measurement and Cumulative Records

  • The Cumulative Recorder:

    • An electromechanical apparatus invented by B. F. Skinner to record real-time operant response patterns continuously.

    • Mechanical operation:

    • A continuous roll of paper unrolls under a recording pen at a constant horizontal speed.

    • Each individual operant response triggers a small vertical upward step of the pen.

    • When the pen reaches the top edge of the recording paper, it automatically resets instantaneously back to the baseline bottom position.

    • Graphical Interpretation:

    • Horizontal Axis: Represents the continuous passage of time.

    • Vertical Axis: Represents the total cumulative number of responses emitted.

    • Distance Between Steps: Corresponds to the inter-response time (IRTIRT); closely spaced steps signify rapid responding, whereas extended flat horizontal segments reflect pauses in behavior.

    • Slope (Slope=ΔResponsesΔTime\text{Slope} = \frac{\Delta \text{Responses}}{\Delta \text{Time}}): Directly quantifies the response rate. A steep slope denotes a high rate of responding; a shallow slope denotes a low rate of responding; a completely horizontal flat line indicates zero responses (complete behavioral pause).


Mechanisms and tracing of a cumulative recorder
  • Modern Data Representation:

    • Behavioral assays utilize automated computer tracking software and video cameras to score response rates directly in discrete time bins (e.g., lever presses per minute across successive training sessions).

Taxonomy of Reinforcement Schedules

  • Definition of Reinforcement Schedule:

    • A rule determining how and when a behavioral consequence (reinforcer) is delivered relative to emitted operant actions.

    • Controls both the overall rate and the micro-architecture (pattern) of responding.

    • The response patterns induced by specific reinforcement schedules are highly conserved across species (from pigeons and rats to human beings) and across diverse classes of reinforcers (food, water, money, points, social praise).

  • Continuous vs. Partial (Intermittent) Reinforcement:

    • Continuous Reinforcement (CRFCRF):

    • Every single emitted response satisfies the contingency and earns a reinforcer.

    • Identical to a Fixed Ratio 1 (FR-1FR\text{-}1) schedule.

    • Ideal for the initial acquisition of a novel operant behavior.

    • Partial (Intermittent) Reinforcement:

    • Only a subset of emitted responses earns a reinforcer; other responses occur without reinforcement.

    • Crucial for maintaining established behaviors and generating resistance to extinction.


Continuous reinforcement timeline showing each response earning a reinforcer
  • The Dimensional Axes of Intermittent Schedules:

    • Basis of Reinforcement Delivery:

    • Ratio Schedules: Response-based. Reinforcer delivery depends strictly on the total number of responses performed by the organism.

    • Interval Schedules: Time-based. Reinforcer delivery depends on the passage of time; a reinforcer is delivered for the first single response emitted after a specified time interval has elapsed.

    • Predictability of Reinforcement Delivery:

    • Fixed Schedules: The rule governing delivery is identical and unchanging across all successive reinforcers.

    • Variable Schedules: The requirement varies from reinforcer to reinforcer around an assigned mathematical mean value.

Ratio Schedules: Fixed Ratio (FR) and Variable Ratio (VR)

  • Fixed Ratio (FRFR) Schedules:

    • Contingency Rule: Reinforcement is presented after a constant, set number of responses (nn) has been completed (designated as FR-nFR\text{-}n).

    • Cumulative Response Profile:

    • Characterized by a distinctive pause-run pattern (or burst-pause pattern).

    • Post-Reinforcement Pause (PRPPRP): A period of behavioral inactivity occurring immediately following the receipt of a reinforcer.

    • Ratio Run: Following the pause, the organism abruptly resumes responding at a high, steady rate until the ratio requirement is satisfied and the next reinforcer is obtained.

    • Influence of Ratio Size:

    • Higher ratio requirements (e.g., FR-15FR\text{-}15 compared to FR-5FR\text{-}5) produce faster response rates during the ratio run itself.

    • Higher ratios systematically produce longer post-reinforcement pauses.

    • Everyday and Applied Examples:

    • Customer loyalty punch cards requiring a set number of purchases (e.g., "Buy 1212, Get 11 Free" frozen yogurt cards).

    • Dialing a complete phone number (entering a fixed sequence of 77 or 1010 digits before the connection is established).

    • Piecework labor payment systems where compensation is paid per fixed batch of assembled goods.


Cumulative record showing the pause-run pattern and post-reinforcement pause of a Fixed Ratio schedule


Cumulative record comparison of Fixed Ratio 5 and Fixed Ratio 15 schedules showing pause and run rate differences
  • Variable Ratio (VRVR) Schedules:

    • Contingency Rule: Reinforcement is delivered following the emission of a variable number of responses, where the average number of required responses equals nn (designated as VR-nVR\text{-}n).

    • Cumulative Response Profile:

    • Produces remarkably high, stable, and consistent rates of responding.

    • Does not generate substantial post-reinforcement pauses because the very next emitted response could potentially yield a reinforcer.

    • Everyday and Applied Examples:

    • Slot machines and casino gambling apparatuses, where payouts occur unpredictably relative to the number of lever pulls or bets placed.

    • Casting a line during recreational fishing, where catching a fish requires an indeterminate number of casts.

    • Social media engagement metrics, where publishing posts yields variable likes, comments, and retweets.

    • Sports performance, such as taking shots in basketball, where a variable number of attempts leads to a successful basket.


Cumulative record of responding under a Variable Ratio schedule

Interval Schedules: Fixed Interval (FI) and Variable Interval (VI)

  • Fixed Interval (FIFI) Schedules:

    • Contingency Rule: The first response emitted after a fixed, unchanging duration of time (tt) has elapsed since the previous reinforcer is reinforced (designated as FI-tFI\text{-}t).

    • Important Clarification: Reinforcers are not delivered automatically when the time elapses; an operant response must be emitted after the interval expires to trigger reinforcer delivery. Extra responses made during the interval have no effect on reinforcer delivery.

    • Cumulative Response Profile:

    • Generates a characteristic scalloped pattern of responding.

    • Responding drops to zero or near zero immediately after reinforcer delivery (post-reinforcement pause).

    • Responding gradually increases as the end of the interval draws near, reaching its maximal rate immediately before reinforcement.

    • Significance in Cognitive Assessment:

    • Fixed interval schedules serve as standard assays for evaluating an organism's interval timing capabilities and subjective estimation of time.

    • Everyday and Applied Examples:

    • Checking a washing machine during laundry cycles: responding (checking) is absent at first, accelerating sharply as the timer reaches the final minutes.

    • Baking in an oven: checking the oven door accelerates as the prescribed baking duration elapses.

    • Periodic examinations in academic settings, where studying behaviors drop to zero after an exam and accelerate sharply as the next scheduled exam date approaches.


Cumulative record of responding on a Fixed Interval schedule showing scalloped patterns
  • Variable Interval (VIVI) Schedules:

    • Contingency Rule: The first response emitted after an unpredictable, variable duration of time has elapsed since the previous reinforcer earns reinforcement, where the average interval length is tt (designated as VI-tVI\text{-}t).

    • Cumulative Response Profile:

    • Generates steady, uniform, moderate-to-low response rates.

    • Devoid of post-reinforcement pauses or scalloped accelerating segments because the timing of availability is completely unpredictable.

    • Everyday and Applied Examples:

    • Waiting for a cable technician during an expansive appointment window (e.g., between 8:00 AM8:00\,\text{AM} and 12:00 PM12:00\,\text{PM}).

    • Calling an automobile repair mechanic to verify if vehicle repairs are completed.

    • A baker or store clerk waiting for customers to enter a retail store throughout the day.

Comparative Schedule Dynamics: Differential Response Rates and Yoked Comparisons

  • Ratio vs. Interval Schedule Rate Disparity:

    • Ratio schedules (FRFR and VRVR) generate substantially higher overall rates of responding than interval schedules (FIFI and VIVI).

    • Both ratio and interval schedules can yield steady patterns (observed in VRVR and VIVI), but the absolute slope of the VRVR schedule is substantially steeper.

  • Mechanisms Driving Rate Differences:

    • Reinforcement of Inter-Response Times (IRTIRTs):

    • An inter-response time (IRTIRT) is the time interval elapsed between two consecutive responses.

    • Ratio Schedules: Directly reinforce short IRTIRTs (bursts of responding). The more rapidly an organism emits responses, the sooner the ratio count is satisfied and the reinforcer delivered.

    • Interval Schedules: Directly reinforce long IRTIRTs (waiting between responses). A subject that responds at a very high rate during the interval does not accelerate reinforcer availability, effectively setting an upper ceiling or reinforcer rate limit.

  • Experimental Verification via Yoked Control Paradigms:

    • Matthews et al. (1977):

    • Evaluated human subjects performing a button-pushing operant task rewarded with points.

    • Compared an unconstrained VRVR subject with a yoked VIVI subject.

    • Yoking Procedure: The exact times at which the VRVR subject completed ratio requirements and earned reinforcers defined the precise interval durations assigned dynamically to the VIVI subject.

    • Outcome: Despite identical numbers and temporal distributions of earned reinforcers, the VRVR participant maintained a much higher response rate than the yoked VIVI participant, proving that response rate differences stem from the underlying reinforcement schedules rather than total reinforcer frequency.


Cumulative records comparing human button pressing on Variable Ratio versus yoked Variable Interval schedules
  • Developmental Timing and Fixed Interval Schedules:

    • Lowe, Beasty & Bentall (1983):

    • Investigated fixed interval responding in human infants using an operant task involving cylinder manipulation for food reinforcers.

    • Compared behavior across different interval parameters: FI-10 sFI\text{-}10\,\text{s}, FI-20 sFI\text{-}20\,\text{s}, and FI-30 sFI\text{-}30\,\text{s}.

    • Findings:

      • Systematic increases in interval duration produced slower overall response rates and longer post-reinforcement pauses.

      • Validated that the basic temporal and rate dynamics of FIFI schedules are functional during early human developmental stages.


Cumulative records of infant cylinder pushing across FI 10, FI 20, and FI 30 schedules