Chapter 9: Extinction of Conditioned Behavior

Overview of Extinction

• Extinction involves a decrease in conditioned responding. This occurs when the conditioned stimulus (CS) or instrumental response is no longer paired with the unconditioned stimulus (US) or reinforcer, respectively, during nonreinforced trials.
• Extinction applies to both classical and instrumental conditioning.
• Extinction is not simply the removal of the CS-US or Response-Outcome (R-O) association. Instead, it's viewed as the acquisition of a second association that interferes with the expression of the original association formed during conditioning.
• This contradicts associative models like Rescorla-Wagner, which propose that extinction training erases the initially acquired association.

Extinction vs. Forgetting

• Extinction differs from forgetting.
• Forgetting occurs when responding declines due to the passage of time, particularly if the CS or instrumental response doesn't occur during that time.
• In extinction, the CS or instrumental response does occur, but without the US or reinforcer.

Effects of Extinction Procedures

• Extinction procedures have two main behavioral effects:
  • Primary effect: A decrease in responding when the response no longer leads to reinforcement.
  • Secondary effect: An increase in response variability.

Neuringer, Kornell, and Olufs (2001) Experiment

• Instrumental learning experiment with rats using two response levers and a round response key.
• During conditioning, rats had to make a sequence of three responses (any lever-key combination) to receive a food reinforcer.
  • Group 1: Varied their responses to get the reinforcer.
  • Group 2: Yoked to Group 1 (received reinforcement based on Group 1's responses).
• After reinforcement, extinction treatment was conducted.
• Results:
  • Extinction decreased responding in both groups.
  • Both groups showed increased response variability.
  • Response sequences likely during conditioning persisted during extinction and were integrated with less frequent responses.
• Conclusion:
  • Extinction alters the rate (decrease) and variability (increase) of behavior, but leaves the underlying structure relatively intact.

Emotional Reactions to Extinction: Frustration

• Extinction procedures can cause strong emotional reactions, specifically frustration.
• Frustration is the emotional response to the withdrawal of an expected reinforcer after the organism has become accustomed to receiving it.
• Effects of frustration:
  • Energizes behavior (the organism responds more intensely).
  • May lead to aggressive behavior (extinction-induced aggression).

Forms of Recovery from Extinction: Spontaneous Recovery

• Spontaneous recovery:
  • Involves the re-emergence of a previously extinguished conditioned response after a period of delay.
  • The recovery is rarely complete; the response is typically weaker than it was prior to extinction.
• Typical Pavlovian conditioning experiment:
  • Two CSs (S1 and S2) and one US.
  • Both CSs are initially paired with the US until they elicit strong conditioned responses.
  • Then, the CSs are presented alone (extinction treatment).
    • Extinction of S1 occurs days before that of S2.
  • Both S1 and S2 are tested at the same time.
    • For S2, test trials begin immediately after extinction trials.
    • For S1, there is a time interval between extinction and testing.
  • Results typically show that responding is stronger to S1 than to S2. The difference in responding at test between the CSs is key.
• Between-group design:
  • Includes two groups: Immediate and Delayed.
  • Both groups initially receive CS-US pairings until the CS produces a strong conditioned response.
  • Then, the CS is presented alone for both groups (extinction treatment).
    • Group Immediate: Tested with the CS immediately after extinction training, eliciting a weak response.
    • Group Delayed: Tested with the CS several days after extinction training, eliciting a stronger response than Group Immediate, demonstrating spontaneous recovery.

Forms of Recovery from Extinction: Renewal

• Renewal is the recovery of an acquired behavior following extinction when the contextual cues present during extinction are removed during testing.
• Bouton and King (1983) Experiment:
  • Used conditioned suppression experiments (fear conditioning) with rats.
    • Rats were trained to press a lever for food.
    • A tone CS was paired with footshock.
  • Classical conditioning treatments were conducted in two different experimental chambers (Context A or B) providing distinct contextual cues.
  • Subjects were divided into three groups:
    • Group NE: Received no extinction training.
    • Group A: Received 20 extinction trials in the original conditioning context (Context A).
    • Group B: Received 20 extinction trials in an alternative context (Context B).
  • Then all subjects were tested in Context A (ABA renewal design).
    • ABA = Acquisition, extinction, and test in Contexts A, B, and A respectively.
  • Results:
    • Group NE demonstrated the strongest suppression.
    • Group A demonstrated the weakest suppression.
    • Group B demonstrated more suppression than Group A.
  • Conclusion:
    • Renewal of fear in Group B suggests that extinction does not reflect unlearning.

Bouton et al. (2011) Experiment

• Used the ABA renewal design, similar to Bouton and King (1983), but with instrumental conditioning.
• Three phases:
  • Acquisition: Lever-pressing is reinforced in Context A.
  • Extinction: Lever-pressing is extinguished in Context B.
  • Test: Conducted in both Context B and Context A.
• Role of the excitatory properties of Context A:
  • Group Exp: Animals received nonreinforced exposure to Context A, to extinguish any possible excitatory properties acquired by this context.
  • Group No Exp: Context A received no additional exposure.

Further Findings on Renewal

• Renewal is not specific to fear conditioning; it has also been found in:
  • Pavlovian appetitive conditioning.
  • Conditioned inhibition.
  • Instrumental conditioning.
• Renewal occurs with contextual cues created by drug states.
• Renewal also occurs if the subject is removed from the extinction context and tested in a completely novel context (ABC and AAB renewal designs).

Explanations of Renewal

• Explanation of most associative models (e.g., Rescorla-Wagner):
  • Summation of the excitatory properties of the context and the residual effects of excitation to the CS.
• Bouton’s (1993) explanation:
  • The memory of extinction is specific to the cues present during the extinction treatment.
  • The memory of extinction depends on its training context for retrieval.
  • The original acquired performance (conditioning) generalizes more to other contexts than the extinction performance.
  • The contextual cues disambiguate the meaning of the CS, indicating whether the CS predicts the US.

Clinical Importance of the Renewal Effect

• Pathological responses (e.g., fear, addiction) are often learned before non-pathological responses (learned during therapy).
• Non-pathological responses are more dependent on the therapy context.
• When the patient encounters the stimulus outside the therapy context, the pathological response can be renewed.
• Treatments to enhance exposure therapy (extinction):
  • Performing therapy in the context where the pathological response was first acquired or in a similar context.
  • Performing therapy in several different contexts to promote generalization of extinction to new contexts.

Forms of Recovery from Extinction: Reinstatement

• Reinstatement refers to the recovery of excitatory responding to an extinguished CS due to US-alone exposures (CS+, then CS-, then + (US-alone) leads to strong responding at test).
• Reinstatement may be a result of contextual conditioning.
• But reinstatement does not seem to be due to summation of contextual conditioning with residual excitation of the CS.
• The context during reinstatement disambiguates the meaning of the CS.
• US-alone presentations in the test context restore the excitatory properties of the contextual cues.
• This allows the contextual cues to reactivate the memory of the excitatory CS-US association.

Forms of Recovery from Extinction: Resurgence

• Resurgence involves the reappearance of an extinguished target response when another reinforced response is extinguished.
• In other words:
  • Phase 1: R1+ trials followed by R1- trials, resulting in conditioning and, then, extinction of R1.
  • Phase 2: R2+ trials followed by R2- trials, resulting in conditioning and, then, extinction of R2 and… the reappearance of R1 (i.e., resurgence).
• Demonstration by Winterbauer and Bouton (2011) involving appetitive operant conditioning with two levers (L1 and L2).
• This effect could explain why, in people with developmental disabilities, problem behaviors (e.g., disruptive or self-injurious behaviors) that had been replaced with functional communications training (FCT) reappear when communication attempts are ignored by the teacher.

Retention of Knowledge of the Reinforcer

• If extinction does not disrupt the CS-US association, then US devaluation should effectively disrupt responding to an extinguished CS.
• Rescorla (1996):
  • Two groups of rats received appetitive conditioning with two CSs (tone or light) and two USs (food and sucrose).
  • Then one group received extinction with both CSs and the other did not.
  • Finally, all subjects had one US devalued by taste aversion conditioning (i.e., US paired with $LiCl$).
  • Results:
    • In both groups, responding to the CS whose US had been devalued was lower than responding to the CS whose US had not been devalued.
  • Conclusions:
    • Extinction leaves the CS-US association intact.
    • Extinction does not reduce sensitivity to the devaluation of the US.
  • Similar results in instrumental learning (Rescorla, 1993).

Enhancing Extinction: Number and Spacing of Extinction Trials

• Number of extinction trials: Increasing the number of extinction trials enhances the effectiveness of extinction treatment.
• Spacing of extinction trials:
  • Cain, Blouin, and Barad (2003): Massed extinction treatment (extinction trials close together) was more effective than spaced extinction treatment (extinction trials spread out).
  • Contrary findings in appetitive conditioning (Moody, Sunsay, & Bouton, 2006).
  • Massed extinction treatment produces more rapid decrement in responding within a session.
  • However, with this treatment responding typically recovers between sessions.

Enhancing Extinction: Immediate versus Delayed Extinction

• Conducting extinction immediately after acquisition training might interfere with consolidation of the memory of acquisition, therefore yielding more robust effects.
• A study by Myers, Ressler, and Davis (2006) showed that conducting extinction of a conditioned fear right after conditioning treatment can result in a more permanent extinction (relative to the effectiveness of extinction when separated from conditioning by a long time interval).
• But Rescorla (2004) found contrary results in appetitive conditioning.
• Also, studies have found that, with immediate extinction, the behavior is more likely to show spontaneous recovery and renewal (Woods & Bouton, 2008; Chang & Maren, 2009).
• The loss of behavior seems to be more enduring when extinction trials are delayed 24 hours after the end of acquisition.

Enhancing Extinction: Repetition of Extinction/Test Cycles

• Alternating periods of rest (retention intervals) and test results in a progressive decline of spontaneous recovery.
• That is: CS+, then CS-, then rest, then CS-, then rest (and so on)… results in a weaker spontaneous recovery on each test.
• Study by Rescorla (2004):
• Although the studies have not been conducted yet, one would expect similar results with renewal, reinstatement, and resurgence.

Enhancing Extinction: Conducting Extinction in Multiple Contexts

• The renewal effect can be reduced by conducting the extinction treatment in multiple contexts:
  • That is: (CS+)A, then (CS-)B, then (CS-)C, then (CS-)D, then (CS-)E, etc.
• Study by Thomas et al. (2009):
  • Four groups of animals received either 36 or 144 extinction trials in either 1 or 3 contexts, following appetitive operant conditioning (lever pressing for a drop of sucrose).
• A fifth group received no extinction at all.
  • Renewal of responding was found in groups 1C-36Ext, 3C-36Ext, and 1C-144Ext.
  • Renewal was eliminated in group 3C-144Ext.
  • Thus, the elimination of renewal required extensive extinction treatment in multiple contexts.

Enhancing Extinction: Presenting Extinction Reminder Cues

• Extinction performance can also be restored by introducing cues that reactivate the memory of extinction.
• This effect has been found in taste-aversion conditioning (Brooks, Palmatier, Garcia, & Johnson, 1999) and in appetitive conditioning (Brooks, 2000; Brooks & Bouton, 1993).
• Basic design:
  • Phase 1: CS+.
  • Phase 2: Cue→CS-.
  • Phase 3: Retention interval.
  • Test: CS vs. Cue→CS.
• The presentation of the cue prior to the CS at test retrieves the memory of extinction, thereby attenuating the impact of the retention interval.
• These cues are referred to as retrieval cues.

Enhancing Extinction: Presenting Extinction Reminder Cues to Reduce Renewal

• As in the case of spontaneous recovery, renewal of responding can also be reduced by introducing a retrieval cue for extinction (Brooks & Bouton, 1994).
• Basic design:
  • Phase 1: (CS+)A.
  • Phase 2: (Cue→CS-)B.
  • Test: (CS)A vs. (Cue→CS)A.
• The presentation of the cue reduces the renewal effect by retrieving the memory of extinction.

Enhancing Extinction: Compounding Extinction Stimuli

• Compounding stimuli during an extinction treatment can also result in enhanced extinction.
• Experiment by Rescorla (2006):
  • Assessed whether, following elemental extinction treatment, additional compound extinction trials further decreased conditioned responding.
  • Basic design:
    • Acquisition: L+, X+, Y+.
    • Element extinction: L-, X-, Y-.
    • Compound extinction: LX-, Y-.
    • Retention interval.
    • Test: X vs. Y.
  • The results showed weaker responding to X than to Y.
    • That is, in comparison to elemental extinction of Y, extinction of X in compound with L resulted in deeper extinction of responding.
  • Similar findings showing that this treatment reduces the impact of reinstatement and the rate of reacquisition of an extinguished stimulus (Rescorla, 2006; Thomas & Ayres, 2004).

Enhancing Extinction: Priming Extinction to Update Memory for Reconsolidation

• Presenting a CS activates the memory of acquisition, rendering it in a modifiable state.
• The memory can be changed before it is reconsolidated and returned to long-term store.
• The period during which the memory can be modified is called the reconsolidation window (short-lived, less than 6 hours).
• The memory of acquisition could be substantially altered by conducting extinction during the reconsolidation window.
• Research in fear conditioning with rats (Monfils et al., 2009) and humans (Schiller et al., 2010) shows that presenting the CS prior to extinction trials produced more robust extinction.
• Schiller et al.:
  • Extinction was comparable with and without priming exposure to the CS.
  • Yet, no spontaneous recovery was found when extinction was conducted following priming exposure (i.e., during reconsolidation window).

What is Learned in Extinction?

• Extinction does not involve the “unlearning” of the CS-US or R-O associations.
• Extinction can be explained as due to the development of an inhibitory S-R association (Rescorla).
• Nonreinforcement of a response in the presence of a specific stimulus produces an inhibitory S-R association.
• This association serves to suppress the response whenever the stimulus is present and is consistent with the renewal effect.
• This hypothesis predicts that extinction is highly dependent on contextual cues.

Why Extinction Training Produces an Inhibitory S-R Association

• Extinction involves a special type of nonreinforcement because extinction follows a history of reinforcement.
• Nonreinforcement is aversive only after a history of appetitive reinforcement.
• Emotional effects of nonreinforcement are dependent on the prior history of the subject.
• Nonreinforcement in the face of expected reward triggers an unconditional aversive frustration reaction.
• This aversive emotion discourages responding during extinction through the establishment of an inhibitory S-R association.

Development of an Inhibitory S-R Association in Instrumental Extinction (Rescorla, 1993)

• Where:
  • $N$ and $L$ = Noise and light discriminative stimuli.
  • Rc = Common response (nose poking) for all subjects.
  • $P$ = Food pellet reinforcer.
    *$R1$ and $R2$ = Lever press and chain pull, counterbalanced across subjects.

• Results:
  • Responding to the stimulus with which the response had been extinguished was significantly weaker than responding to the alternate stimulus.
• This indicates that an inhibitory S-R association was established to a specific stimulus and response.
• These results, together with the results of an increasing number of studies, lead to the conclusion that extinction and conditioned inhibition might be based on the same processes.

Paradoxical Reward Effects: Overtraining and Magnitude Reinforcement

• Overtraining extinction effect:
  • More extensive reinforcement training produces more rapid extinction.
  • This is paradoxical because it represents fewer responses in extinction after more extensive reinforcement.
• Magnitude reinforcement extinction effect:
  • Responding declines more rapidly in extinction following reinforcement with a larger reinforcer.
  • Nonreinforcement is more frustrative if the individual has come to expect a large reward than if the individual has come to expect a small reward.

Paradoxical Reward Effects: Partial Reinforcement Extinction Effect (PREE)

• Partial reinforcement extinction effect (PREE):
  • Slower extinction and fewer frustration reactions when the organism has been on an intermittent schedule of reinforcement (partial reinforcement).
  • The schedule of reinforcement determines the level of behavioral and emotional effects of extinction.
  • PREE tends to produce a persistence in responding during extinction.
    • Daily-life examples:
      • Gamblers continue to gamble after long strings of loses.
      • A child will badger a parent for candy after the parent says “no” several times.
  • PREE has also been demonstrated in Pavlovian conditioning.
  • PREE occurs in the same subjects if they experience continuous reinforcement for one set of cues and intermittent reinforcement for another set of cues.

Mechanisms of the Partial Reinforcement Extinction Effect: Discrimination Hypothesis

• The discrimination hypothesis:
  • Explains the PREE by assuming that extinction is easier to detect (discriminate) after continuous reinforcement (CRF) than after partial reinforcement (PRF).
• Jenkins (1962) and Theios (1962):
  • Two groups of animals.
    • Phase 1:
      • Group 1 = PRF.
      • Group 2 = CRF.
    • Phase 2:
      • Groups 1 and 2 = CRF.
    • Phase 3:
      • Groups 1 and 2 = Extinction.
  • The discrimination hypothesis predicts that both should extinguish at the same rate.
  • Results:
    • Group 1 responded more than Group 2 during extinction.
      *Persistence of responding does not come from greater difficulty in detecting the start of extinction. Something long-lasting is learned during partial reinforcement.
• Partial reinforcement teaches subjects not to give up in the face of “failure” (nonreinforcement) and that times of “success” (reinforcement) do not mean that “failure” will never occur again.

What is Learned from Partial Reinforcement?: Frustration Theory and Sequential Theory

• Two accounts: Frustration theory and sequential theory.

Frustration theory (Amsel, 1958):

• Persistence in extinction results from learning to continue responding when the animal expects to be nonreinforced (or frustrated).
• Intermittent reinforcement results in learning to respond in the face of nonreinforcement.
• There are different stages during the course of learning-to-persist:
  • First, there is a conflict since both types of trials (reinforced and nonreinforced) are occurring.
  • As training continues, the instrumental response ends up being reinforced when the subject expects nonreinforcement.
  • Thus, the instrumental response becomes conditioned to the expectation of nonreward.
  • In the end, the intermittent reinforcement results in learning to make the instrumental response as a reaction to the expectation of nonreward.

Sequential theory (Capaldi, 1967):

• This theory assumes that subjects remember both recent reward and nonreward trials.
• The memory of nonreward becomes a cue for performing the instrumental response.
• The explanations/predictions of this theory depend greatly on the sequence of reward (R) and nonreward (N) trials.
• Consider the sequence: RNNRRNR.
• The fourth and last trials (underlined) are critical because this is when the subject is reinforced in the face of the memory of nonreward.
• With enough of these experiences the subject learns to respond whenever nonreward occurs.

Comparison of Frustration Theory and Sequential Theory

• Both frustration theory and sequential theory provide effective explanations for PREE.
• Memory mechanisms make more of a contribution when training trials are close together.
• Emotional learning mechanisms make more of a contribution when trials are spaced out.

Resistance to Change and Behavioral Momentum

• Behavioral momentum is based on the concept of momentum in Newtonian physics (Nevin, 1992).
• That is, the momentum of a physical object is the product of its weight (or mass) and its speed (e.g., a bullet or a train).
• The behavioral momentum hypothesis predicts that behavior that has a great deal of momentum will also be hard to stop or disrupt.
• Behavioral momentum has been examined using multiple schedules of reinforcement and a variety of sources of disruption. Examples of sources of disruption:
  • Adding food before training.
  • Adding food during intervals between components of multiple schedules of reinforcement.
  • Extinction.

Dube and McIlvane (2001):

Subjects:

*Two individuals with severe mental retardation.,

Multiple schedule:

*Two different computerized instrumental tasks (A and B) with food as the reinforcer.

• Task A = Touch a white square with a black cross that appeared in one of the corners of the computer screen.
• Task B = Touch one of two pictures that was the same as a picture that was briefly presented at the start of the trial.
  • The tasks were alternated in blocks of 15 trials (ABAB or BABA).

Phase 1:

• Task A was continuously reinforced (CRF) and Task B was reinforced on a PRF (VR 4) schedule.
• During momentum testing, the same schedules of reinforcement were in effect.
• However, extra food was given before the session and between task components.
• Additionally, there were other distractors such as cartoons, toys, and various sound effects.

Results:

• The introduction of distractors reduced responding.
• More important, the disruption was greater for Task B (PRF, or VR 4 schedule) than for Task A (CRF schedule).
• This suggests that PRF responding had less momentum than CRF responding.

Phase 2:

• Each component of the multiple schedule was reinforced on a CRF schedule.
• The results of this phase showed that both components of the multiple schedule were equally disrupted.

Phase 3:

• Identical to Phase 1: Reintroduced the PRF (VR 4) schedule.
• Replicated the results of Phase 1.

Conclusions:

• Behavioral momentum is directly related to the rate of reinforcement.
  • More reinforcement = Higher behavioral momentum.
• Behavioral momentum is unrelated to the rate of responding.
  • The rate of responding was comparable in both CRF and PRF schedules.
• The importance of reinforcement rate rather than response rate suggests behavioral momentum is due to Pavlovian (S-O) associations, instead of instrumental (R-O) associations.

Positive and Negative Aspects of Behavioral Momentum Theory

Positive aspects of behavioral momentum theory:

• It is a single concept that tries to explain behavioral persistence.
• It applies well to sources of disruption that cause changes in performance rather than ones that produce new learning.

Negative aspects of behavioral momentum theory:

• It can not explain PREE.
• It has difficulties with disruptive manipulations such as extinction.