Unit 4: Learning from reward and prediction

Why do we learn from rewards?

We learn which behaviors lead to good outcomes and use that experience to guide future decisions.

What is reinforcement?

The process by which a consequence makes a behavior more likely to happen again.

What is a reward prediction?

An expectation that a rewarding outcome will happen in the future.

After getting a better grade, you start using practice questions more often. What process does this show?

Reinforcement. The better grade made you more likely to use practice questions again.

Before your next exam, you expect practice questions to help you get a better grade. What does this show?

Reward prediction. You expect the same study strategy to lead to a good outcome again.

What do we learn in Pavlovian conditioning?

We learn that a cue predicts an outcome. For example, a rat learns that a tone signals that food is coming.

How does a neutral cue become a predictor of reward?

When the cue is repeatedly followed by a reward, the animal learns to expect the reward when the cue appears.

What is an anticipatory response?

A response that happens before the reward arrives. For example, a rat hears the tone and approaches the food dish.

What do we learn in instrumental conditioning?

We learn that an action produces an outcome. For example, a rat learns that pressing a lever gives it food.

What happens to an action when it is reinforced?

It becomes more likely to happen again. The rat presses the lever more often because pressing it has led to food.

How can you tell Pavlovian and instrumental conditioning apart?

Pavlovian asks, “What cue tells me food is coming?” Instrumental asks, “What action can I take to get food?”

Why do unexpected outcomes lead to more learning?

They give us new information that can change what we expect next time.

Why do fully predicted outcomes lead to less learning?

The outcome matches what we already expected, so there’s little new information.

What happens in Phase 1 of the tone-and-light example?

The animal learns that the tone predicts food.

What happens in Phase 2?

The tone and a new light appear together, followed by the same food reward.

Why does the animal learn little or nothing about the light?

The tone already predicts the food, so the light adds no new information. This is called blocking.

What determines how much we learn from a reward?

How much the reward differs from what we expected. A surprising reward gives us more reason to update our prediction.

Which two brain regions contain neurons that produce dopamine?

The ventral tegmental area (VTA) and the substantia nigra pars compacta (SNc).

Where does the nigrostriatal pathway go, and what does it do?

It goes from the SNc to the dorsal striatum (DS). It supports voluntary movement, habit formation, and motor learning.

Where does the mesolimbic pathway go, and what does it do?

It goes from the VTA to the ventral striatum (VS), including the nucleus accumbens (NAc). It helps us learn which cues predict rewards and motivates us to seek them.

Where does the mesocortical pathway go, and what does it do?

It goes from the VTA to the prefrontal cortex (PFC). It supports executive functions like planning, helps us act toward goals, and influences emotional responses.

A rat learns that a tone predicts food. Which dopamine pathway is most directly involved?

The mesolimbic pathway, because it helps the rat learn the connection between a cue and a reward

How does dopamine help guide future actions?

It helps us learn from rewards and use that experience to decide what to do next.

What is a reward prediction error?

The difference between the reward you received and the reward you expected. In the formula =R-V, R is the reward received and V is the expected reward.

What happens to dopamine firing when an unexpected reward arrives?

Dopamine firing spikes when the reward arrives. This is a positive prediction error because the outcome was better than expected.

What happens to the dopamine response once a cue reliably predicts a reward?

The spike shifts from the reward to the cue. When the expected reward arrives, there’s little or no additional spike because it matches the prediction.

What happens if the cue appears but the expected reward never arrives?

Dopamine firing still spikes at the cue, then dips below baseline when the reward should have arrived. The dip signals a negative prediction error because the outcome was worse than expected.

When is the reward prediction error zero?

When the reward received exactly matches the reward expected.

What is reward prediction error (δ), and how is it calculated?

The difference between the reward received and the expected reward: δ = R − V.

What equation is used to update expectations?

V_new = V_old + αδ.

What do V_new, V_old, and α represent?

V_new: updated expectation;

V_old: previous expectation;

α: learning rate, ranging from 0 to 1.

What does the learning rate determine?

How much we change our expectations after an unexpected outcome.

How do high and low learning rates affect expectations?

High: larger, faster updates.

Low: smaller, gradual updates.

If you expect 80 points and receive 95, what is the prediction error?

δ = 95 − 80 = +15; the outcome was better than expected.

With an expectation of 80, a prediction error of +15, and α = 0.5, what is your new expectation?

80 + (0.5 × 15) = 87.5 points.

How do previous experience and confidence influence learning rates?

More experience and confidence mean less influence from a single unexpected outcome; less experience means greater weight on new information.

How does environmental stability influence learning rates?

Consistent outcomes: update expectations gradually. Changing outcomes: update expectations quickly.

What does the graph show happens when you repeatedly receive 95 points?

Your expected score gradually approaches 95 points.

What is temporal difference learning?

Updating predictions across time using cues, before the final reward occurs.

What is the temporal difference prediction error equation?

δₜ = rₜ + γVₜ₊₁ − Vₜ

What do the terms in the equation mean?

• δₜ: prediction error now 

• rₜ: reward received now (0 if none yet) 

• Vₜ: expected value now 

• Vₜ₊₁: expected value of the next state 

• γ: discount factor (0–1); how much future rewards matter

How does the discount factor affect future rewards?.

Higher γ: future rewards matter more.
Lower γ: future rewards matter less

When do dopamine neurons increase firing early in learning?

When the unexpected reward arrives, producing a positive prediction error (δ > 0).

How does dopamine firing change after learning that a tone predicts reward?

Firing increases at the tone, with little additional response when the predicted reward arrives.

How does temporal difference learning explain this shift?

The cue predicts a future reward, so the dopamine response shifts from the reward to the earlier predictive cue.