Clinical Psychology Lecture 7

Assessment

The evaluation or estimation of the nature, quality, or ability of someone or something 

1. Does someone “have” depression? Which measure does the best job of detecting it? 

2. Is a person competent to stand trial? 

3. Was a child abused by his or her parent? 

4. How well do radiologists detect tumors or other abnormalities? 

5. How well do we identify terrorists? 

6. How well do lie-detector tests work? 

7. Can we tell when someone is deliberately presenting themselves as struggling more psychologically than they actually are? 

Prediction

Saying or estimating that (a specific thing) will happen in the future or will be a consequence of something 

1. How well can we predict suicide/violence? 

2. How well do we predict weather, earthquakes, hurricanes, tornados, etc. 

3. How well can we predict whether a bank will fail? 

4. Can you predict whether job or school applicants will succeed on the basis of admissions materials? 

5. How well do various aptitude tests predict success? 

Common assessment & prediction goals in psychology

Screening: Identification of those experiencing clinically significant difficulties 

Diagnosis/care formulation: Specification of DSM clinical diagnosis and conceptualization of clinically significant difficulties 

Prognosis/prediction: Estimation of the likelihood of clinically significant difficulties 

Treatment monitoring and evaluation: Examination of how well prevention or intervention is working 

Fundamental questions — Are A&P approaches:

Standardized? Unique aspects of testing situation and assessor minimized 

a. Provision of comparable materials/items across assessors, detailed instructions re: administration, detailed descriptions of scoring procedures 

Reliable? Internal consistency, test-retest, inter-rater 

Valid? Content, concurrent, predictive, convergent, discriminant, incremental 

Useful? Practical utility of approach, relative to others 

Alcohol Use Disorders Identification Test (AUDIT)

Developed by World Health Organization (WHO) to screen current alcohol problems in last year 

Best-available brief screener for alcohol dependence, alcohol abuse, high-risk drinking that doesn’t meet criteria 

Used around the world in variety of contexts 

AUDIT domains

1. Alcohol consumption (1-3) 

2. Drinking behavior/dependence (4-6) 

3. Alcohol-related problems (7-10) 

Scoring: 

a. Qs 1-8 scored 1, 2, 3, 4 

b. Qs 9-10 scores 0, 2, or 4 

c. Total scores range from 0-40 

Recommended cutoff  ≥ 8 for identification of potential alcohol problems in adults 

AUDIT is standardized

AUDIT is reliable: 

Internal consistency: Consistency of responses across items on measure (Cronbach’s alpha: .81) 

Test-retest reliability: Consistency of responses over time (r = .91) 

Inter-rater reliability: Consistency of judgments across raters (kappa = NA) 

AUDIT is valid

Content validity: Extent to which item content reflects concept of interest (three domains covered well) 

Concurrent validity: Association of measure with another presumably relate measure at same point in time (AUDIT correlates with current psychosocial problems) 

Predictive validity: Association of measure with another presumably related measure at future point in time (AUDIT correlates with future psychosocial problems) 

Convergent validity: Moderate-to-strong association of measures intended to measure same of similar concept (AUDIT correlates strongly with other alcohol measures) 

Discriminant validity: Weaker association with measure intended to assess different concept (AUDIT correlates more weakly with personality characteristics than with alcohol measures) 

Incremental validity: Extent to which measure predicts more than what already could predict (AUDIT predicts future alcohol problems better than any other self-report measures) 

AUDIT is useful

Low cost & brief screening measure that is more accurate than other similar measures 

Sensitivity

Ability of test to correctly identify those with disorder (or assessed/predicted concept); proportion of those with disorder/condition who are correctly identified 

Specificity

Ability of test to correctly identify those without disorder (or assessed/predicted concept); proportion of those without disorder/condition who are correctly identified 

We want both sensitivity and specificity to be high (typically .80 or above) 

Cutoff value

Determines whether A&P device makes positive or negative assessment/prediction 

AUDIT cutoff selection

As sensitivity increases, specificity decreases 

Sensitivity and specificity depend on the cutoff 

a. Higher cutoff (more conservation): Fewer people are given diagnosis – decreased sensitivity, increased specificity

b. Lower cutoff (more liberal): More people are given diagnosis – increased sensitivity, decreased specificity

Cutoff selection depends on what we value more: (a) correctly detecting disorder (increases TPs), or (b) not classifying those without disorder as disordered (decreases FPs) 

Costs of associated with cutoff selection:  

a. Cost of low sensitivity: Fewer people get treatment who need it (or we don’t detect bomb or don’t send guilt person to jail) 

b. Cost of low specificity: People receive treatments who don’t need it and may receive costly treatments with serious side effects (or we send innocent person to jail) 

Is AUDIT accurate?

AUDIT-based classifications of alcohol disorder related to interview-based classifications 

What cutoff value use for A&P purposes? 

Most use 8 in practice; for college students, 6 more appropriate 

a. Sensitivity = .78, Specificity = .57 

b. % with disorder correctly identified = 78% 

c. % without disorder correctly identified = 57% 

d. % without disorder incorrectly identified as having disorder = 1.0 -- specificity = 43% 

Using cutoff of 6 means that more persons with alcohol problems will be identified as having problems.  

Using cutoff of 8 means that you will wrongly identify persons as having problems less frequently 

Evidence-based assessment

Need to provide information on (in)accuracy of assessments, as well as psychometric properties 

a. Don’t overstate limits of A&P powers 

b. Recognize biases and boundaries of competence 

Clinical (completely unstructured)

Clinician selects, measures, and combines risk factors and produces risk estimate solely according to clinical experience and judgment 

a. Easy to imagine how could be far less reliable and valid 

Actuarial/statistical (completely structured)

Risk assessment completely determined by prediction equation and established cutoff scores 

Base rate

Proportion of people who meet criterion 

Base rates of violent recidivism typically very low 

Less frequent events (suicide, violence) are much harder to predict; predictions tend to be less accurate, very high rates of false positives 

In clinical psychology, predictions are often made for things that have low base rates 

a. We often overstate our ability to predict low-frequently events 

True positive

Person with disorder/condition classified by screen/questionnaire as having disorder/condition

False positive

Person without disorder/condition classified by screen/questionnaire as having disorder/condition

True negative

Person without disorder/condition classified by screen/questionnaire as not having disorder/condition

False negative

Person with disorder/condition classified by screen/questionnaire as not having disorder/condition

Prediction equation

Provides an overall risk score from a weighted combination of risk factors