Reinforcement Learning Notes

Classical Conditioning

• Unconditioned Stimulus (US) → Unconditioned Response (UR)

  • Example: Food (US) → Salivation (UR); Shock (US) → Withdrawal (UR)

• Conditioned Stimulus (CS) + US → UR

  • Example: Tone (CS) + Food (US) → Salivation (UR); Mild Touch (CS) + Shock (US) → Withdrawal (UR)

• CS alone → Conditioned Response (CR)

• A neutral stimulus (CS) becomes associated with a reflex.

Reinforcement Learning: Maximizing Reward

• Animals learn to obtain rewards and avoid punishments.

• Reinforcement learning theory provides a formal model for this learning.

• Animals often perform unrewarded preparatory actions to obtain later rewards.

  • Example: A mouse leaving its burrow to search for food.

Reinforcement Learning: Key Topics

• Theoretical approach.

• Brain circuits underlying RL.

• Dopamine cells: Neural basis for RL.

• RL and prediction errors.

• Mechanisms that make this kind of learning possible.

Core Principle of Reinforcement Learning

• Learning which actions lead to positive and negative outcomes.

• Animal is rewarded for a correct outcome.

• The animal may not know which action led to the reward (especially with a large action set or a sequence of actions).

• Mistakes provide no specific information about the error.

• Reinforcement learning involves exploration followed by selection through reward.

• Operant conditioning is a form of associative learning relevant to reinforcement learning.

Operant (Instrumental) Conditioning

• Response → Stimulus (reward)

  • Example: Lever press → Food

• Conditioned response (CR) → Conditioned stimulus → (reward, food)

  • Example: Lever press → reward (food)

• Trial/error

  • Accidental behaviour can lead to reward.

  • Repetition of behavior leading to reward leads to association between the reward and the behaviour.

The Law of Effect (Thorndike, 1911)

• Responses closely followed by satisfaction are more likely to recur in the same situation.

• Coming back for more.

• The strength of association is important in associative learning.

Learning to Predict Reward

• Predict reward $R$ given stimulus $U=1$. Goal: to predict the reward $V=wU$

• Need to learn to change the connection between the CS ($U$) and the US (reward).

• $\Delta w = \varepsilon(R-V)U$

• After learning of consistent pairing: V=R

• Associative learning occurs not because two events co-occur but because that co-occurrence is unanticipated on the basis of current associative strength.

  • $U$ = CS availability

  • $V$ = reward prediction

  • $R$ = reward availability

  • $w$ = weight of the connection between U and V (strength of association between CS and US)

  • $\varepsilon$ = learning rate

  • $R-V$ = prediction error

• Rescorla-Wagner rule (1972)

• Reward prediction error.

Brain Circuits and Reinforcement Learning

• Hypothetical circuit model of reinforcement learning

• Basal Ganglia: Striatum, Pallidum, Substantia nigra , Subthalamic nucleus

• Doya K. (2008) Nature Neuroscience 11, 410 - 416

Intracranial Self-Stimulation

• Rat learns to press a lever to receive electrical stimulation in the substantia nigra from dopamine neuron.

• Lever-pressing rate increases with stimulus intensity until it reaches optimal stimulation.

• Dopaminergic pathways are important in reinforcement learning.

• Reynolds et al. (2001) Nature 413, 67-70 (Dopamine as a substitute for reinforcement).

Dopaminergic Cells and Reward Prediction

• Recordings from dopaminergic cells in monkeys during conditioning.

• Start of training: Reward elicits firing.

• After learning: Cell fires at the CS (conditioned stimulus)

• Reward does not appear when predicted: Firing is suppressed (negative prediction error).

• Continued lack of reward leads to extinction of the CR.

• Schultz (1998) J. Neurophysiol.

Dopamine Response Progression

• Dopamine cell fires less after the reward as the animal learns about the CS.

• Hollerman & Schultz (1998) Nat. Neurosci.

• Population response of DA cells.

  • Top: light – no response; drop of juice - strong response

  • Middle: after learning strong resp. to reward predicting trigger but not to predicted reward

  • Bottom: resp. to a cue preceding the trigger by 1s in a spatial learning task

Coding of Reward Probability

• Monkeys conditioned with visual stimuli indicating reward probability.

• Response probability at the stimulus position increased with reward probability.

• At probabilities 0.25-0.75 firing of the dopamine neurons occur at both the CS and reward position.

• If CS is not a reliable predictor of reward, then the dopamine cell responds both at the time of the CS and at the time of the US.

• Fiorillo et al. (2003) Science

Sensitivity of Dopamine Cells to Reward Level

• Behavioral and neuronal responses to conditioned stimuli increase with expected reward value.

• Tobler et al. (2005) Science

Learning Driven by Predictive Error

• Connections between stimulus and response are enhanced when they receive a positive predictive reward error and are weakened when they receive a negative predictive reward error.

Blocking

• Learning - acquisition of reliable predictions about future outcomes.

• Once an outcome can be reliably predicted, then no further learning occurs.

• Blocking studies: Animal associates stimulus (e.g. noise) with US (e.g. shock). Then a second stimulus (e.g. light) is added. Animal doesn't learn that the light also has predictive power because it can already predict the outcome.

• Errors in predicting the outcome of the reward that leads to learning and not the reward itself.

• If the reward level is raised or lowered, then there is a predictive error and learning does occur.

Dopamine Cells and Blocking

• Dopamine cells show blocking in a conditioning task which implies that these cells carry a signal related to the prediction error that is used for acquisition.

• A monkey is given a reward after viewing stimulus A, but not B. X is then paired with A, and Y with B. Both pairings are rewarded. After the pairing, the monkey has associated Y with the reward, but X has been blocked and the monkey ignores X presented by itself.

• Waelti et al (2001) Nature

Prediction Error Signals and Dopamine Cells

• Dopamine cells are good candidates for carriers of prediction error signals during reinforcement learning.

• They act as critics telling the cortex how to improve its performance

• They know:

  • When a reward is expected and their response drops if no reward appears at that time.

  • The size of the expected reward. Cells respond more when the reward is bigger than expected and are inhibited when it is smaller than expected.

  • The temporal details of a sequence. A monkey is well trained to receive juice after it has viewed a short sequence of stimuli which appear at expected times.

• Normally, the dopamine cells respond to the first temporally unpredicted stimulus, but not to the rest.

• If one of the middle items of the sequence is omitted, the cells' firing drops at the time when the stimulus should have appeared. Conversely, the cell fires if a stimulus appears prematurely in the sequence.

Brain Areas and Neurotransmitters

• DA, dopamine

• 5-HT, serotonin

• NE, norepinephrine

• ACC, anterior cingulate cortex

• DLPFC, dorsolateral prefrontal cortex

• DS, dorsal striatum

• OFC, orbitofrontal cortex

• VS, ventral striatum

• Doya K. (2008) Nature Neuroscience 11, 410 - 416

Summary of Reinforcement Learning

• Reinforcement learning is broadly a process of exploration followed by selection through reward.

• Associative learning occurs not because two events co-occur but because that co-occurrence is unanticipated. This is explained by the Wagner – Rescola model.

• Learning is driven by a predictive error signal, the difference between the actual reward and the expected reward.

• Blocking experiments show the importance of reward prediction error both behaviourally and on the level of DA cell signalling.

• DA cells can predict the probability and the magnitude of the reward.

• The dynamic interaction of brain areas composing the cortex–basal ganglia loops, as well as other subcortical structures, especially the amygdala, is believed to result in reward-dependent selection of particular actions.

Learning Outcomes

• Explain the theory of reward learning.

• Interpret the theory by using experimental examples.

• Describe the role of reward prediction error.

• Interpret the role of dopamine in reinforcement and reward learning.

• Explain dopamine signalling during and after reinforcement learning.

• Give examples of how different brain areas can modify reinforcement.