3 - instrumental/operant learning

Dopamine spiral

Instrumental (operant) conditioning

Learning the consequences of your behaviour. action = outcome.

Giant African pouched rats

In Mozambique, giant rats trained to detect explosives in the ground because they are trained to receive a reward when they signal its presence. 20 mins for rats, 25 hours for humans to find

Thorndike's law of effect

Tested ability of cats to escape a box and obtain a food reward - time to escape was reduced trial after trial as they learned which behaviour needed to escape. Any behaviour that is followed by a pleasant experience is likely to be repeated, and vice versa.

B.F. Skinner's operant conditioning

Refined Thorndike's law of effect - introduced free-operant trials (animals can operate the apparatus freely)

The Skinner box

Performance of an action (operant e.g. lever press, nose poke) results in reward delivery (reinforcer), increasing the probability that the action will be repeated. the action is 'instrumental' - in producing the consequence

Discriminative stimulus

In both discrete and free operant conditions, individuals can learn to discriminate conditions in which a specific action/response leads to an outcome.

Discriminative stimulus (SD) -> response (R)

If the outcome is pleasant then the frequency of responding/actions may increase in the future when in precense of the discriminative stimulus

In thorndike's box, what are the discriminative stimulus, response, and outcome?

SD - Being inside the box

R - Action lever and ropes

O - Escape and eat food

In skinner's box, what are the discriminative stimulus, response, and outcome?

SD - Lever

R- lever press

O - Reward delivery

Positive and negative reinforcement

Positive reinforcement - gain desirable reward = encouraged behaviour

Negative reinforcement - Remove aversive stimulus = encourage behaviour

Positive and negative punishment

Positive punishment - addition of aversive stimulus = discourage behaviour

Negative punishment (omission) = removal of something pleasant to discourage behaviour

examples of negative/positive punishment/reinforcement

positive reinforcement -ADD GOOD you give a duck a grape and its tasty. he is more likely to swim up to you next time

negative reinforcement - REMOVE BAD a goose is chasing the duck, so he hides underwater to escape. by removing the scary threat, he is more likely to hide underwater next time

positive punishment - ADD BAD a duck tries to steal grapes from a goose. the goose hisses at him. because aggression was added, the duck tries to stop stealing grapes

negative punishment - REMOVE GOOD a duck quacks at you angrily for grapes so you leave. because you took away the tasty grape, the duck learns that aggressive quacking makes tasty treat leave

Primary reinforcer

biologically relevant positive event, such as food, water or sleep. Organisms repeat behaviours that give access to them. Needs no learning to be rewarding.

Negative contrast

The identity of the reward changes behaviour, so a lower-value reward gets a weaker response than a better one

Examples of negative contrast

Babies - suck on an artificial nipple for different liquids. Plain water is almost as preferred as sweetened, but when given sweetened then plain, rate of sucking decreases.

Capuchin monkeys - reject 'unequal pay' when a neighbouring monkey is given a better reward (a grape) instead of both being given cucumbers. they start throwing that shi only when they realise other monkey got it better

Rewards are judged against alternatives, not just by their absolute value!

Secondary (conditioned) reinforcer

A reinforcer with no biological value of its own that has been paired with or predicts a primary reinforcer. For humans, money!!!

Horse or dolphin trainers often pair a clicker/whistle sound with their primary reinforcers (oat or fish), then this is used to reinforce specific behaviours

Conditioned reinforcers teach new responses

Rat nosepoke give a light (CS) plus a food pellet (O). Then an active lever press (R) gives the light only with no food. The light alone reinforces a brand new response

Punishers

Things with negative biological impact - instrumental learning can help animals to avoid these. In humans - fines, jail time, social disapproval.

Punishers vs reinforcers

Behavioural effect of punishers are less reliable in eliciting tpes of action. There are 1. variable behaviours across individuals 2. Discriminative stimulus can encourage cheating (police car/speeding) 3. concurrent reinforcers undermine punishment 4. initially low intensity can prevent learning

Concurrent reinforcement undermines punishment

rats press lever for sucrose or cocaine. introduction of footshocks (punishment) at the same time. rats taking sucrose completely stop lever pressing - a fraction of cocaine seeking taking rats keep responding

ratio schedules and interval schedules

fixed interval - reinforcement delivered at predictable time intervals

variable interval - delivered at unpdredcitable time intervals

fixed ratio - delivered after a predictable number of responses

variable ratio - delivered after an unpredictable number of responses

progressive ratio - effort needed for reinforcement increases

fixed interval

delivered at predictable time intervals.

moderate response rate with significant pauses after reinforcement

e.g. - controlled pain relief

variable interval

delivered at unpredictable time intervals

moderate yet steady response rate

e.g. - checking facebook

fixed ratio

reinforcement delivered after a predictable number of responses

high response rate with pauses after reinforcement

e.g. - piecework in a factory

variable ratio

reinforcement is delivered after an unpredictable number of responses

high and steady response rate

e.g. - gambling

order the reinforcement schedules in order of most cumulative responses to least

variable ratio, fixed ratio, variable interval, fixed interval

Concurrent reinforcement schedules can be used to assess choice behaviour

Pigeon pecking at disk a on a reinforced 2-min schedule. Disk b is a 1-min schedules. Maximum benefits will happen on disk b so the pigeon pecks over twice as much

Behavioural economics

studies how organisms allocate their time and resources, can be used to quantify motivation associated with instrumental behaviour - consumers will allocate resources to obtain maximum subjective value - involves dopamine

mechanisms of reinforcement in the brain

mesolimbic pathway - for craving reward : ventral tegmental area sends dopamine through medial forebrain bundle (MFB) into the nucleus accumbens/ventral striatum.

mesocortical - for planning to get it: vta sends dopamine to prefrontal cortex

nigrostriatal pathway - for movement towards reward: susbtantia nigra sends dopamine into the dorsal striatum

Choice between intracranial self-stimulation and food

rats with intracranial electrodes stimulating the medial forebrain bundle (MFB) preferred to lever press for electrical stimulation even at the expense of losing weight

Intracranial self-stimulation

Rats will press a lever to self stimulate the medial forebrain bundle (MFB) or ventral tegmental area (VTA), which release dopamine in striatal regions such as the Nucleus Accumbens. The ventral striatal dopamine acts as a reinforcer but this evidence is only a correlation

Dopamine in choice behaviour

Rats usually prefer sugar to chow pellets - blocking dopamine (SKF 83566, D1 and raclopride, D2) makes them choose free chow vs lever pressing for sugar pellets.

Rats also choose to gamble when given the choice to do so - blocking dopamine with antagonist Flupoenthixol (D1/D2) reduced it

Dopamine and discriminative stimulus (SD)

Dopamine in the nucleus accumbens may underlie the ability of an SD (a cue signalling reward) to increase pursuit of reward

Types of learning

Perceptual - the ability to learn and recognise stimuli that have been perceived before

Motor - the ability to learn a movement, or coordinate a series of movements

Stimulus-response - involved in the formation of habits

Morris Water Maze

Spatial learning - rats must find a platform relying on visual cues - this improves with learning. Bilateral hippocampal lesions decrease latency to find platform.

Dorsal damage - severe spatial learning deficits

Ventral damage - spares performance

Basal ganglia and motor skill learning

Motor skill learning depends on the basal ganglia (dorsolateral caudate putamen) and dopamine transmission there

Habitual behaviour

Same actions repeated, automatic, insensitive to devaluation of the reward - less regulated by cognitive/inhibitory control. Defined by stimulus-response relationship. Habits are defined by which stimuli elicit behaviours, rather than how behaviours are performed

Outcome devaluation (not habitual)

Tests for habitual vs goal directed behaviours. If behaviour governed by goal-directed Action-Outcome (A-O) relationships, then devaluing the reward will decrease pursuit of the reward. think - the hungry rats

Habit

Behaviour triggered by a situation (stimulus-response, S-R), not by wanting the reward. If you make the reward unwanted (e.g. a rat is fed until full) and the behaviour carries on, it was a habit.

Outcome devaluation in rats

Rats trained to lever press for food on random interval task for 120 or 360 trials. They were then tested either in low or high levels of hunger - rewards was devalued for low hunger rats.

120 condition - reward devaluation decreased lever pressing

360 condition - lever pressing was a higher level in all rats regardless of reward devaluation

---- Satiated rats now lever press like hungry rats - their behaviour is habitual, not goal-directed

Dual-process theory of overtraining

The overall rate of instrumental performance at any given stage of training will be determined by the total strength produced by the summing of the strengths of the A - O and S - R processes

High-fat diet and habits

A high fat diet may be driven by stimulus-response behaviour rather than action-outcome behaviour when given on a Random Interval (RI) schedule. The brain is very sensitive to leptin/insulin. When you eat high fat food and the Nucleus Accumbens is flooded with dopamine, the NA and DMS dopamine receptors are desensitised, also quietening down the prefrontal cortex which helps to make decisions. Instead, the DLS neurons are flooded with glutamate and experience enhanced dopamine transmission from the midbrain, working with the sensorimotor cortex to coordinate 'unconscious' habits.

Dopamien spiral

Information physically moves up a gradient in the brain,

When an action is brand new, the mesolimbic pathway fires dopamine into the ventral striatum to signal desire, which recruits the prefrontal cortex and dorsomedial striatum (DMS) to consciously figure out the action-outcome strategy.

As you repeat the action over and over, the dopamine signal physically migrates sideways through interconnected loops into the substantia nigra and the dorsolateral striatum (DLS).