Psyc 251 - decision making

problem

What to choose out of all the options available to us

importance

Our choices define who we are and how we interact with the world

challenge

What are the options? What are the outcomes and how likely are they? What do we value?

prescriptive approach (expected value)

how should people decide?

expected value (rational choice)

Average outcome if a scenario is repeated many times, calculated using probabilities and values of possible outcomes

advantages to using expected value method

Clear prescription for "correct" choices, leads people, on average, to maximize monetary gains given what they know about the world, keeps peoples decisions internally consistent

disadvantages to using expected value method

Difficult to apply for non-monetary decisions, doesn't explain actual choices by people

descriptive approach (prospect theory)

how do people deicide?

prospect theory (Kahneman and Tversky, 1979)

People do not make decisions based on expected values, probabilities, and absolute outcomes. People make decisions based on subjective utility, decision weights, and relative outcomes (Kahneman: only cognitive psychologist to win the Nobel Prize (2002, in economics))

subjective utility

People transform objective value into subjective utility (utility = usefulness or desirability of an outcome)

Diminishing marginal utility

Subjective utility increases more slowly than objective value, especially at large values, $10 is subjectively worth twice as much as $5, but 10 million is not subjectively worth twice as much as 5 million (where it levels off can be different for different people)

Loss aversion

Losses loom larger than gains, losing $20 feels worse than winning $20 feels good, individual differences in sensitivity to loss

Decision weight

People transform objective probability into subjective decision weights (Small probabilities (but greater than 0%) are overweighted, large probabilities (but less than 100%) are underweighted)

Framing effect (reference dependence)

People make decisions based on gains and losses relative to a point of reference, not based on absolute outcomes, changing the way a question is asked to create a different point of reference leads to different valuations and thus different choices

Prospect theory - people make decisions based on individual:

Subjective utilities (diminishing marginal utility and loss aversion), decision weights (underweight large probabilities and overweight small probabilities), relative outcomes (reference dependence, gain and loss framing)

fourfold pattern of risk attitudes - risk seeking

low probability gain and high probability loss

fourfold pattern of risk attitudes - risk averse

low probability loss and high probability gain

Reinforcement learning

We perform actions in the world and experience the resulting outcomes as good (reward) or bad (lack of reward or punishment)

Old idea about dopamine and rewards

Midbrain dopamine system signals pleasure/reward. however, midbrain dopamine system does not simply signal reward

Learning driven by rewards - Schultz, Dayan and Montague (1997)

Single unit recordings from monkeys midbrain dopamine neurons in ventral tegmental area

Learning driven by rewards - Schultz, Dayan and Montague (1997) conclusions

Activity of midbrain dopamine neurons is related to reward, but dopamine neurons do more than simply report occurrence of reward, they code deviations from predictions about time and magnitude of reward

reward prediction error (RPE)

Reward prediction error (RPE) = actual reward - expected reward (RPE > 0, better than expected. RPE = 0, as expected, RPE > 0, worse than expected)

Dopamine and reinforcement learning

We are continuously predicting expected future reward, we take actions to maximize future reward, when we receive information that violates our expectations, it generates a reward prediction error, as a result, we update our predictions, which may alter our actions

Dopamine pathways in human brain

Starts at ventral tegmental area, midbrain dopamine neurons project to basal ganglia, prefrontal cortex, and many other areas. Can train motor, oculomotor (eyes), executive/associative, emotional/motivation control

reinforcement learning and addiction

Opioids physiologically trigger release of dopamine, this is misinterpreted as a reward prediction error signal, thus, opioids "hijack" the reinforcement learning mechanism (can also happen in gambling)

Reinforcement learning and AI

Many successful systems of last decade combine: Reinforcement learning + neural networks = deep reinforcement learning

Iowa gambling task

Goal: win as much money as possible, two decks are bad (if you keep picking you will lose money) and two are good (if you keep picking you will win money). Bad decks you win 100, good decks you win 50. Bad decks - large gain and large loss, good decks - small gains and small losses

Iowa gambling task results

Control group: picked "good" decks more often (especially near end), shifted to mostly "good" decks before they can explain why, larger SCR response when about to pick from bad deck. VMPFC patients: picked "bad" decks more often (kept picking bad decks even at end), could explain what was happening with decks, but kept picking bad decks, had very small SCR response (little anxiety) about picking bad deck (no predicted feelings of anxiety about picking bad deck)

VMPFC

ventromedial prefrontal cortex (or orbital frontal cortex)

Iowa gambling task conclusions

Conceptual knowledge alone did not lead to good decision making, predictions of expected emotions (somatic markers) were necessary for good decision making, expected emotions preceded conceptual understanding, patients with damage to VMPFC could not generate expected emotions, patients overemphasized immediate reward over long-term outcomes (temporal discounting)

functions of prefrontal cortex

Maintenance and updating of goals, inhibition of prepotent actions, shifting between rules, sets and tasks, monitoring and adjusting performance, integrating multiple sources of value

Prefrontal cortex

frontal cortex anterior to pre-central gyrus

Rostral/caudal

abstraction

Rostal PFC

Complex, abstract, long timeframe

Caudal PFC

Simple, concrete, short timeframe

Ventral/dorsal

what and why vs where and how

Dorsal PFC

Where, how, action oriented

Ventral PFC

What, why, meaning oriented

Medial/lateral

emotion vs cognition

Medial PFC

hot/affective/motivation, value-based, internal/body-oriented

lateral PFC

cold/cognitive/rules, feature-based, external/enviorment-oriented

WHACH model of PFC organization

what-how, abstraction, cold/hot (O'Reilly, 2010)