Clinical PSY/ Lecture 7: Assessment Overview

What is Screening?

Identification of those experiencing clinically significant difficulties

What is Diagnosis/Case Formulation?

Specification of DSM clinical diagnosis and conceptualization of clinically significant difficulties

What is Prognosis/Prediction?

Estimation of the likelihood of clinically significant difficulties

What is Treatment Monitoring and Evaluation?

Examination of how well prevention or intervention is working

What are the three types of reliability that are evaluated for assessment and prediction approaches?

Internal consistency, Test-retest, and Inter-rater reliability

Suppose that a researcher develops a new interview measure of obsessions and compulsions. The researcher arranges for three clinicians to administer the new clinical interview to each of 50 clients and computes a kappa statistic. What form of reliability is the researcher evaluating for the new measure?

inter-rater reliability

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure, a well-established measure of the same concept, and a quality-of-life measure to a large sample of adults in the US at the same timepoint. The researcher finds that the new measure correlates more strongly with the quality-of-life measure than the well-established measure of obsessions and compulsions. What form of validity is the researcher establishing for the new measure?

Incremental validity

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure twice (2 months apart) to a large sample of adults in the US. If the researcher examines the correlation between the scores on the new measure at the two time points, what form of reliability is the researcher evaluating for the new measure?

Test-retest

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure, the best available measure of obsessions and compulsions, and a measure of quality of life to a large sample of adults in the US. The researcher finds that scores on the new measure correlate more weakly with the quality of life measure than with the well-established measure. What form of validity is the researcher evaluating for the new measure by comparing these correlations?

Discriminant validity

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure to a large sample of adults in the US. If the researcher computers Cronbach's alpha for the new measure, what form of reliability is the researcher evaluating for the new measure?

Internal consistency

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure to a large sample of adults in the US at one timepoint and then a measure of quality of life to the same sample four months later. If the researcher correlates scores on the two measures, what form of validity is the researcher evaluating for the new measure?

Predictive validity

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure and the best available measure of obsessions and compulsions to a large sample of adults in the US. If the researcher correlates scores on the two measures, what form of validity is the researcher evaluating for the new measure?

Convergent validity

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The researcher then administers the new measure and a measure of quality of life to a large sample of adults in the US. If the researcher correlates scores on the two measures, what form of validity is the researcher evaluating for the new measure?

Concurrent validity

Suppose that a researcher develops a new self-report measure of obsessions and compulsions. The research consults with numerous experts in Obsessive and Compulsive Disorder when developing items to ensure that all aspects of the concept of Obsessive and Compulsive Disorder are included in the items on the new measure. What form of validity is the researcher establishing for the new measure?

Content validity

What is the AUDIT?

The best-available brief screener for alcohol abuse, alcohol dependence, and high-risk drinking (Alcohol Use Disorders Identification Test)

What is Content Validity?

Extent to which item content reflects concept of interest

What is Concurrent Validity?

Association of measure with another presumably related measure at SAME point in time

What is Predictive Validity?

Association of measure with another presumably related measure at FUTURE point in time

What is Convergent Validity?

Moderate-to-strong association of measures intended to measure same or similar concept

What is Discriminant Validity?

Weaker association with measure intended to assess different concept

What is Incremental Validity?

Extent to which measure predicts more than what already could predict

What does the Sensitivity of an assessment/prediction approach tell us?

Proportion of those with a disorder or condition who are correctly identified

What does the Specificity of an assessment/prediction approach tell us?

Proportion of those without a disorder/condition who are correctly identified

What does it mean if a measure of Bulimia shows a Sensitivity value of .60?

60% of those with Bulimia are correctly identified as Bulimic by the measure

What does it mean if a measure of Bulimia shows a Specificity value of .80?

80% of those without Bulimia are correctly identified as not having Bulimia by the measure

If a measure of Bulimia shows a Sensitivity value of .60, what percentage of people with Bulimia are NOT identified as Bulimic by the measure?

40%

If a measure of Bulimia shows a Specificity value of .80, what percentage of people without Bulimia are WRONGLY identified as Bulimic by the measure?

20%

What does a Cutoff Value determine for an assessment/prediction approach?

Whether the approach makes a positive or negative assessment or prediction

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. If you increase the clinical cutoff fromm 20 to 24, will MORE or FEWER people receive a Social Anxiety Disorder diagnosis?

FEWER

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. If you increase the clinical cutoff fromm 20 to 24, will the measure's sensitivity INCREASE or DECREASE?

DECREASE

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. If you increase the clinical cutoff fromm 20 to 24, will the measure's SPECIFICITY INCREASE or DECREASE?

INCREASE

What is a True Positive?

Person with disorder/condition classified by screen/questionnaire as having disorder/condition

What is a False Positive?

Person without disorder/condition classified by screen/questionnaire as having disorder/condition

What is a True Negative?

Person without disorder/condition classified by screen/questionnaire as not having disorder/condition

What is a False Negative?

Person with disorder/condition classified by screen/questionnaire as not having disorder/condition

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. What do we call those who are correctly identified by the measure as meeting diagnostic criteria for Social Anxiety Disorder?

True Positives

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. What do we call those who are correctly identified by the measure as not meeting diagnostic criteria for Social Anxiety Disorder?

True Negatives

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. What do we call those who are wrongly identified by the measure as meeting diagnostic criteria for Social Anxiety Disorder?

False Positives

Suppose that you administer a measure of Social Anxiety to 1000 college students, where higher scores are associated with a greater likelihood that the person would be diagnosed with Social Anxiety Disorder in a comprehensive clinical interview. What do we call those who are wrongly identified by the measure as not meeting diagnostic criteria for Social Anxiety Disorder?

False Negatives

Does Specificity increase or decrease as Sensitivity increases?

Decrease

What is the cost of low sensitivity for a measure that aims to predict whether someone makes a suicide attempt?

Missing a lot of people who will make a suicide attempt

What is the cost of low specificity for a measure that aims to predict whether someone makes a suicide attempt?

Wrongly predicting a lot of people will make a suicide attempt

What does a Prediction Equation provide us?

An overvall risk score from a weighted combination of risk factors

What does it mean for an assessment/prediction appraoch to be completedly clinical?

Clinician selects, measures, and combines risk factors and produces risk estimate solely according to clinical experience and judgment

What does it mean for an assessment/prediction nappraoch to be completedly actuarial or statistical?

Risk assessment completely determined by prediction equation and established cutoff scores

What does it mean that the Violance Risk Appraisal Guide (VRAG) shows a sensitivity value of .65?

The proportion of those correctly predicted by the VRAG to show violent recidivism

What does it mean that the Violance Risk Appraisal Guide (VRAG) shows a specificity value of .65?

The proportion of those correctly predicted by the VRAG not to show violent recidivism?

Are clinical or actuarial methods more accurate at making clinically relevant predictions?

Actuarial methods

What are Base Rates?

The proportion of people who have a particular condition or meet a particular set of criteria

When Base Rates are low, do our assessments/predictions tend to be MORE or LESS accurate?

LESS