5: R-O Relations

Instrumental Learning/ Conditioning

• synonymous to R-O Learning

• learning in which the probability of a given behavior is altered by its consequences

  • behavior is instrumental in producing the outcome

Opernat Learning/ Conditioning

subset of instrumental learning

• instrumental learning in which the organism specifically learns to manipulate its environment to achieve a desired outcome

Instrumental Criteria

1. the behavioral modification depends on a form of plasticity

2. the modification depends on the organism’s experiential history

3. (a) the modification outlasts (extends beyond) the environmental contingencies used to induce it (b) the experience has a lasting effect on performance

4. imposing a temporal relationship between a response and an outcome alters the response

Additional Operant Criteria

1. The nature of the behavioral change is not constrained (ex. either an increase or decrease in the response can be established)

2. the nature of the reinforcer is not constrained (a variety of outcomes can be used to produce the behavioral deficit)

discrete-trial method

a form of instrumental learning where an organism may only perform a behavior to earn a response only at specific times

• ex. iclicker quizzes

free-operant method

a form of instrumental learning where an organism may perform a behavior to earn a response at any time

• ex. checking social media

the type of instrumental learning depends on two factors

• nature of the reinforcer

• relation to the behavior

reward learning (positive reinforcement)

behavior leads to an appetitive outcome (relates to nature of the reinforcer)

• increases likelihood of behavior

Punishment learning (positive punishment)

behavior leads to an aversive outcome (relates to nature of the reinforcer)

• decreases likelihood of behavior

omission learning (negative punishment)

behavior removes an appetitive outcome (relates to relation to the behavior)

• decreases likelihood of behavior

escape/avoidance learning (negative reinforcement)

behavior removes an aversive outcome (relates to relation to the behavior)

• increases likelihood of behavior

Instrumental Learning in the Lab

• reward: bar press → food

• omission: bar press → no food

• punishment: bar press → shock

recall (preparedness in instrumental learning)

learning occurs in a heterogenous behavioral substrate

• some action-outcome relationships are learned more easily than others

• Key-peck → Grain: A pigeon naturally pecks at food, so learning that pecking produces grain is relatively easy.

• Treadle → Turn off shock: A pigeon can learn to step on a pedal to stop an unpleasant stimulus.

Heterogeneous behavioral substrate

Animals aren't born as blank slates. They have natural behaviors and instincts that influence what they learn easily

recall (contingency in instrumental learning)

extra US’s undermine Pavlovian conditioning

extra outcomes disrupt instrumental conditioning

schedules of reinforcement

1. continuous reinforcement (CRF)

2. fixed ratio (FRX)

3. variable ratio (VRX/RRX)

4. fixed interval (FIX)

5. variable intervals (VIX)

continuous reinforcement (CRF)

stable 1:1 ratio of behavior to reinforcer

• behavior always produes reinforcer

fixed ratio (FRX)

stable X:1 ratio of behavior to reinforcer

• X number of responses always produces the reinforcer

variable ratio (VRX/RRX)

average of X:1 ratio of behavior to reinforcer

• the number of responses needed to produce the reinforcer will vary, but average out to X

fixed interval (FIX)

stable interval separates rewardable behaviors

• behavior will only produce reinforcer after a constant amount of time has passed

variable interval (VIX)

average interval separates rewardable behaviors

• behavior will only produce reinforcer after a variable amount of time has passed

learned helplessness

idea that unavoidable aversive events impair performance on future avoidable aversive events

• Experiment (Maier, Overmier, Seligman)

  • three groups: Master (escapable shock), Yoke (inescapable shock), and unshocked

learned helplessness: signaled active avoidance

• master, yoked, and unshocked groups and 24 hours later put in a test shuttle avoidance where they could turn to the other side of the box to avoid an uncontrollable shock

• yoked group took the longest to respond when it came to the shock, as they previously learned that they could not control the outcome

• Master: Experienced control → better avoidance learning.

• Yoked: Experienced no control → impaired avoidance learning.

• Unshocked: No previous shock experience → normal avoidance learning.

learned helplessness hypothesis: causes motivational and associative deficit

• related to Julian Rotter’s Locus of Control

  • internal: “I believe I can control my own future.”

  • external: “My life is directed by forces outside of my control.”

• contributes to common psychiatric disorders, including anxiety, depression, and PTSD

learned helplessness: behavioral therapy

behavioral therapy for after learned helplessness is learned

• rats underwent two sessions of shock: restrained-restrained (RR), inescapable shock-escapable shock (IE), inescapable shock-inescapable shock (II), inescapable shock-restrained (IR)

• after 24 hours, rats were put in a test shuttle avoidance

• II and IR rats took longer to respond to move to another side of the box

• IE and RR took less time to move

learned helplessness: behavioral immunization

giving an animal experience with control over its environment BEFORE exposing it to uncontrollable negative situations

• rats underwent two sessions of shock: restrained-restrained (RR), escapable shock-inescapable shock (EI), inescapable shock-inescapable shock (II), restrained-inescapable shock(IR)

• after 24 hours, rats were put in a test shuttle avoidance

• II and RI rats took longer to respond to move to another side of the box

• EI and RR took less time to move

learned helplessness: endogenous opioids

learned helplessness can be conditioned in most mammals

• used endogenous opioids: chemical naturally produced by your body that help reduce pain

• used naltrexone: an opioid antagonist, meaning it blocks opioid receptors and prevents opioids from having their usually effects

left graph: latency to respond when performance when FR2 was required

• yoked animals took the most time to respond, then yoked+naltrexone rats

• shows that Blocking opioid receptors with naltrexone reduced the learned helplessness effect.

right graph: pain reactivity

• The inescapable shock group (I) showed increased tail-flick latency, suggesting reduced pain sensitivity. When naltrexone was given, this effect decreased.

• This suggests that uncontrollable shocks activate the body's natural opioid system, producing pain relief (analgesia).

organisms are always behaving in some way

ex. dog training:

• dog sits on command

  • receives treat immediately

    • dog associates sitting w/ treat

  • dog poops in house while owner is away

    • dog walks around, eats food, sleeps

    • dog greets owner upon return, owner yells at dog about poop

      • dog learns the wrong lesson

secondary (conditional) reinforcers

a Pavlovian association is formed between a neural stimulus and a primary reinforcer

• Gold Stars in classrooms

marking stimuli

brief auditory or visual cue distinguishing target response from other behaviors

• not valued

• ex. shaping activity

exceptions to the importance of contiguity

• secondary (conditional) reinforcers

• marking stimuli

instrumental behavior typically begins as goal-directed and becomes habitual over time

to test whether a behavior is goal-directed, we can use reinforcer-devaluation

• same as US devaluation

• establish RO-contingency → devalue outcome → test behavior

• if habitual, rats press the same as rats without devaluation (control)

• if goal-directed, rats press less than control rats

ration schedules produce more responding than interval schedules

• feedback functions: how quickly the organism learns to maximize benefit

  • internal feedback functions: reach an asymptote earlier than ratio feedback functions

concurrent schedules of reinforcement

organism has two response options that each lead to reinforcement, but on different schedules

example:

• Lever A yields a large pile of food, but is on a 10-minute fixed-interval schedule

• Lever B yields only a single food pellet, but is on a 5:1 fixed ratio schedule

matching law

IF similar effort is required for response alternatives AND

IF switching from one alternative to the other is easy…

THEN the distribution of responses will depend only on schedule of reinforcement for each response.

THUS, the relative rate of each response is equal to the relative rate of reinforcement obtained w/ that response alternative

generalized matching law

takes differences in effort, reinforcer strength, and switching from one response to the other

• if the initial conditions of matching law are not met, it doesn’t work

concurrent-chain schedules

lock the organism’s initial choice in for the remainder of a trial

• choice link: initial decision wherein participant can select either option

• terminal link: trials following initial trial where only selected option is available

drive-reduction theory

traditional view:

• reinforcers satisfy biological needs

  • mediated by internal drives

• Clark Hull: drive reduction provides a way to derive the internal drives of a species

problems w/ drive-reduction theory

• long and implausible list of drive

  • ex. monkeys will respond to watch a toy train

• learning can occur w/o drive reduction

  • ex. rats learn to seek saccharine = sweet, but no nutritional value

premack principle

given two responses arranged in an operant conditioning procedure, the more probable response will reinforce the less likely behavior, but the less probable response will not reinforce the more probable behavior

view: reinforcing events are opportunites to engage in behavior

• rate would lever press for opportunity to sniff

• rat would not groom for opportunity to sniff

Premack vs. Hull

• premack didn’t designate a special class of events as reinforcing

• Hull:

  • If a rat is thirsty, drinking reduces its thirst.

  • If a rat is deprived of water, drinking becomes more valuable because it satisfies that biological need.

• Premack:

  • something you WANT to do more can be used as a reward for something you WANT to do less

problems w/ premack’s approach

its difficult to quantify time spent performing individual behaviors

• ex. how to quantify drinking behavior?

• 
  

response deprivation theory

proposed a schedule of reinforcement must deprive the organism of the consequent reinforcer

• is not fixated on the time spent performing a response

  • avoids measurement problems

premacks theory in simple terms

a high-probability behavior reinforces a low-probability behavior

• focuses on which behavior you prefer more

response deprivation theory in simple terms

a behavior becomes reinforcing when access to it is restricted below its normal level

• focuses on how much access to a behavior you are allowed

bliss point

ideal amount of time an organism would naturally like to spend doing different activities when it has free access to them

• perfect balance between activities

backward bending labor supply curve

People initially work more as wages rise, but eventually may work less because they can afford more leisure

early neurological view

believed that all learning occurred via common mechanism

modern neurological view

multiple systems underlie different types of memory

early in training for a t-maze

place learning (spatial/explicit)

late in training for a t-maze:

response learning (habit/impliicit)

• lesioning the hippocampus disrupts place learning

  • facilitates habit learning

• lesioning the basal ganglia disrupts habit learning

basal ganglia

striatum:

• globus pallidus

• ventral pallidum

substantia nigra:

• subthalamic nucleus

• ventrolateral nucleus of the thalamus

striatum involvement in learning

• involved in motivation, reinforcement, and planning movements

• received dopaminergic input from the substantia nigra

  • 2 key receptors

substantia nigra involvement in learning

• produces dopamine

• neurons synapse onto neurons in the striatum

amygdala

basolateral nucleus encodes reward magnitude and valence

striatum

integrates positive and negative with behavior over multiple trials

habit learning is incremental

more trials = stronger habit

in complex situations, organisms need to do more than represent reward

must also consider how much reward/punishment will be gained now vs. in the future

orbital frontal cortex (OFC)

allows the organism to weigh the relative value of alternative outcomes

• value: estimate of relative gain both now and in the future

  • lesions to the OFC:

    • trouble incorporating negative feedback from previous training

    • more dependent on amygdala = more impulsive

    • trouble learning that reinforcement contingencies have changed

delayed discounting

can be modelled in both animals and humans

• in rats, lesions of OFC bias behavior toward immediate reward

• in humans, choosing the immediate reward is associated w/ increased limbic activity (amygdala)

  • choosing the delayed reward elicits OFC activity

• people w/ substance-use disorders tend to be biased towards immediate rewards

  • fail to weigh long-term negative consequences

two classes of striatal projection neuron

distinguished by type of dopamine receptor (D1 vs D2)

D1 receptor expressing direct pathway

involved in reward learning and initiating action

D2 receptor expressing indirect pathway

involved in punishment learning and inhibiting action

direct vs indirect pathways experiment

set up: used two groups of rats to implant w/ mechanism to stimulate either direct pathway neurons (D1 medium spiny neurons; dMSN) or indirect pathways neurons (D2 medium spiny neurons; iMSN)

• both groups of rats were presented w/ 2 levers

  • lever 1: engaged stimulation

  • lever 2: no stimulation

findings:

• direct pathway mice worked for stimulation

• indirect pathway mice avoided stimulation

conclusion: direct pathway encodes reward, indirect pathway controls punishment, both involve dopamine

conditional place preference experiment

animals placed on 1 side of a divided chamber and received stimulation

• direct pathway rats preferred the stimulation side during re-exposure

• indirect pathways rats avoided the stimulation side during the re-exposure