PSYC 406 Decision Making

Statistical Determinants of Decision
Quality

validity of tests and measures, base rate of characteristics, selection rate

base rate (BR)

what proportion of the population possesses relevant characteristics (ex. proportion of kids benefitting from learning disability classes)

selection rate

percentage of the population selected (ex. kids in learning disability classes)

cutting score

score used to divide test scores into predictions about who has characteristic or not

hits

correct predictions

misses

incorrect predictions

true positive

phenomenon is predicted and it occurs

true negative

phenomenon is not predicted and it does not occur

false positive

phenomenon is predicted but it does not occur

false negative

phenomenon is not predicted to occur but it does occur

total hit rate equation

= true positives + true negatives / n

test sensitivity

probability test result is positive when phenomenon is present

test sensitivity equation

= true positives / true positives + false negatives

test specificity

probability the test result is negative when phenomenon is not present

test specificity equation

= true negative / true negatives + false positives

when BR is lower than SR what happens?

more false positives (better to be conservative)

decision making with low validity

best to go with a decision based on base rates than developing test unless the number of false positives is important (ex. drug has dangerous side effects)

what is the % of correct decisions dependent on?

SR, BR, test validity

when base rate is higher than selection rate

more false negatives (better to be liberal)

where to get base rate information?

available info, historical records, build database to collect info over time

clinical assessment components

tests(IQ, personality, neuropsychology), behavioral assessment, clinical/environmental info, clinical interview (structured and unstructured)

phases in decision making

data collection, data integration and prediction

data collection phase of decision making

mechanical scores(ex. questionnaires), judgemental scores (interview)

data integration phase and prediction

clinical (integrating material, writing report with recommendation), statistical (combining information statistically)

myth of experience

beyond certain amounts of training, more experience does not translate into more accurate diagnoses

myth of more information

more info does not necessarily give more accurate predictions

myth of configuration/patterns

clinicians' decisions can be modeled using a formula

Goldberg (1965)

compared accurate decisions from humans vs statistical combinations; training between staff and trainees did not have an effect (accuracy was 62%); all statistical procedures did better than the clinicians' average

Clinician error 1: overweight of positive instances

people remember true positives more and other data is ignored: false positives and base rate,

Clinician error 2: similarity heuristic

individuals predict future behavior that resembles test info (eg. tells violent stories ->assault); ignores base rate and validity (can still be correct if BR is very high and test info is valid)

three reasons clinicians aren't better at prediction

overweight successful prediction/forget incorrect predictions, ignore BR, don't use all available information

how to improve clinical judgment?

systematically consider alternatives, collect feedback about decisions/predictions, think about statistical prediction issues

what do clinicians do well?

provide input into statistical models, generate hypotheses, provide predictions when no formula (yet) exists