Psychological Testing Exam 1

Constructs

A concept that is inferred from observed patterns in responses (Behavioral, affective, cognitive)

Hypothetical, abstract variables that we cannot necessarily see, but we infer thru observations

Understanding them offers us insight into something interesting and/or important abt the human experience

Conceptual versus Measured Variables

Conceptual: Can be conceptualized but not precisely measured

• Subject to different interpretations; defintiional clarity is crucial

Measured: Concrete measures or procedures used to quantify constructs

• Should be highly specific and resistant to misinterpretation

Assumptions of Measurement

• Constructs exist, and can be measured

• Measurement is imperfect

• All measurement tools have strengths and weaknesses

• Multiple approaches to measurement is preferable to a single one

• Tests performance can generalize to non-test performance

• Tests can be fair and helpful

Psychological testing

The administration, scoring, and interpretation of measurement tools designed to quantify psychological constructs

Psychometrics

The scientific discipline concerned w the theory and construction of psychological tests; provides standards by which to judge the effectiveness of measurement approaches

Barnum Statements

General statements that can technically apply to anyone. Individuals give high accuracy ratings to descriptions that are “tailored” to them when in reality it fils a large population

Test Types (Typical Response sets and their forms, Maximum Performance tests and their forms)

Objective Personality Tests

• Ex, Self-report in which test-taker answers items on a Likert scale

Projective Personality Tests

• Test taker looks at ambiguous stimulus, is prompted to say something about the stimulus, clinician interprets their story

  • Rorshach Ink blot test

Achievement tests: Assess knowledge/skills in an area in which the test-taker has received instruction

• Ex, Speed: Timed spelling test with many (usually easy) words

• Ex, Power: Spelling bee (Words get increasingly harder)

Aptitude tests: Assess knowledge or skills acquired from life experience

• Ex, Speed: How many figures can you accurately code in 20 seconds?

• Ex, Power: Solve visual-spatial problems of increasing difficulty

Ceiling/Floor effects & their implications for measurement

The clustering of individual survey responses around a certain value

• Ceiling effect occurs when a considerable % of participants score the best of maximum possible score, floor effects occur when the opposite happens

Creates a distribution that would not be considered normally distributed and generates an average value so close to the limits of the scale

• Limits observed variability of the responses

• Errors of measurement

  • Scale construction limitations, sample selection issues, content issues in the variable of interest

Standardization sample (norm-referenced vs. criterion referenced claims)

A large, representative, group of people who provide norms for a test

• Allows you to be able to make norm-referenced claims (interpretation of performance is compared to the performance of other people)

  • Ex, Scored better than 93% of same aged-peers

• Without standardization sample, you are restricted to making criterion-referenced claims

  • Ex, Got a score of 93%

Variable types

Property of an object/event that can take on different observed values

Construct/Conceptual Variable — Can be conceptualized but not precisely measured

• Subject to different interpretations; definitional clarity is crucial 

Operational/Measured Variable — Concrete measures or procedures used to quantify constructs 

• Should be highly specific and resistant to misinterpretation

Discrete: A variable that takes on distinct, countable values (Y or N)

Continuous: A variable that takes on any value within a range, and the number of possible values within that range is infinite

Independent Variable: Manipulated variable, lvls of IV

Dependent Variable: Outcome variable

Measures of central tendency and variability

Mean: x/n. Average of the scores

Mode: The score that shows up the most often

Median: The score in the middle

Variability — The spread of cluster of a set of data

• Range: Difference between the highest and lowest values

• IQR: The range of the middle half of a distribution (Q3-Q1)

• Variance: Average of squared distances from the mean

• Standard Deviation: Average distance from the mean

Normal distribution

When the normal curve has a mean of zero and a standard deviation of 1

Z-Scores, standardization

A standardized measure of how far a particular data value is from the mean of a normally distributed data set

Percentiles

Score greater than a percentage of all scores in a sample

• Ex, A score in the xth percetile would be above approximately x% of all scores in its sample

Correlation

A statistical measure that expresses the extent to which two variables are linearly related (changing at a constant rate). Correlation NOT causation

T-Test

Used to test whether the difference between the response of two groups is statistically significant or not.

ANOVA

Analysis of Variance that compare the means of two or more groups by analyzing variance. Compares the amount of variation between the group means to the amount of variations within each group

Chi-Square

A statistical hypothesis test used in the analysis of contingency tables when the sample sizes are large. Used to examine whether two categorical variables are independent in influencing the test statistic

Evaluates whether distributions of categorical variables differ between groups

Reliability

Degree to which the measure is consistent (Consistency)

Classical Test Theory

Predicts outcomes of psychological testing such as the difficulty of items, precision of estimates, or the ability of test-takers. Based on the idea that a person’s observed/obtained score on a test is the sum of a true score.

• Xi = T + E

  • Xi = Test/measure score, T = The true value/level the individual is on the construct, E = Error

Random vs. Systemic Error

Random error — By chance. Chance difference between the observed and true values of something

• Ex, Researcher misreading a weighing scale records an incorrect measurement)

Systemic error — Test taking conditions. Consistent or proportional difference between the observed and true values of something.

• Ex, A miscalibrated scale consistently registers weight as higher than they actually are

Random error and rules of zero ?

• Mean of random error is zero

• Zero correlation between T and E

• Zero correlation between E across measurements

Primary sources of error (Content sampling, time sampling, errors)

Content Sampling error — variability that can enter into test scores as a result of chance factors related to the content of the specific items included in a test

Time sampling — variability inherent in test scores as a function of the fact that they are obtained at one point in time rather than at another

Scoring/administrative error — Improper administration/execution of a survey.

Measuring reliability (test-retest, alternative forms, inter-rater, split half, internal consistency)

Test-retest — Administering the same test on two separate occasions to one or more groups of individuals

Alternate-form — One measure, and then another version that is parallel but distinct. A correlation test will then determine the relationship between the two tests.

• Intended to estimate the amount of error in test scores that is attributable to content sampling error

Inter-rater — Having multiple people evaluate a score (Two independent evaluators)

Split-Half Reliability — Administer a test to a group of individuals and to create two scores for each person by splitting the test into halves

• Loses statistical power and deflates the actual relationship bewteen the two

• Formula 2(rhalf)/1+rhalf

Internal Consistency — Cuts the test into possible split halves, and then averaging them all

• Is an alpha value

  • Excellent: Greater than .9

  • Good: Greater than/Equal to .7, Less than .9

  • Acceptable: Greater than/equal to .6, Less than .7

  • Poor: Greater than/equal to .5, Less than .6

  • Unacceptable: Less than .5

• Statistical procedures deisnged to assess the extent of inconsistency across test items

Standard Error of Measurement (How to determine prescribed confidence intervals for scores)

The Standard Deviation of the distribution of scores that would be obtained by a person if they took an infinite # of alternate forms of a test comprised of item randomly sampled from a test domain.

Confidence Intervals (Look at the exercises on the slide show)

What is validity?

Degree to which the measure actually reflects what it is intended to measure

Types of validity and how they’re established

Face — Does the measure look like it’s measure the concept we intent to measure?

Content — Is the test fully representative of what it aims to measure/How well are we “covering” the construct?

Criterion-related — Do the results accurately measure the concrete outcome they are designed to measure?

Construct — Does the test measure the concept that it’s intended to measure?

Convergent — When a measure is related in an expected way (either positively or negatively) with some other measure

Divergent — When a measure is not related to an outcome that it is not expected to be associated with

Factor analysis (What does it tell us and how is it related to validity)

Statistical approach designed to investigate whether a number of items are related to a smaller number of unobservables (ex, latent) factors

• Exploratory — Discover and test structure of a construct

• Confirmatory — Tests a predefined model

Allows us to know what the underlying structure of our construct is → Helps us determines whether our measure is matching the a priori/theoretically informed structure we expected

Tells us which items load onto each of the subdomains and if any items are not meaningfully clustering with other items

How do we establish criterion-related validity? What does it mean when it’s concurrent vs. predictive?

To establish this validity, the test results need to be compared w criterion variables/gold star measurements (Other tests that are widely accepted as valid measures of a construct).

Concurrent = Measures tests and criterion variables in the present

Predictive = Measures those in the future

Convergent/Discriminant/Contrasting groups

Convergent — When a measure is related in an expected way (positive/negative) w/ some other measure (correlation expected)

Discriminant — This measure has some precision that is only for this measure and not others. Two unexpected constructs when a measure is not related to an outcome that is not expected to be associated w?

• Relationship not expected, no correlation

Contrasting — Do groups differ in expected ways on the measure?

Sensitivity/Specificity

Sensitivity — Proportion of patients w the condition that test positive

• 100% means test correctly identifies everyone w the condition

• Most certain test result is negative (Don’t have it)

  • Ex, Social Anxiety Disorder (Y/N afraid of public speaking)

    • High sensitivity, Low Specificity. You are getting a majority of SAD individuals, but also a lot of non-SAD individuals

Specificity — Proportion of patients w/o the condition that test negative

• 100% means test correctly identifies everyone without the condition

• Most certain test results is positive (DO have it)

  • When you have a tool that is highly specific, it will miss when someone does have a conditiona. bunch

What are validity pitfalls in test construction (And strategies to avoid this)

Construct underrepresentation — When our measurement fails to identify core components

• Measure needs to capture the proper domain

Construct-irrelevant variance — When the measure identifies components/constructs that are unrelated to the proper domain

• Make sure reading lvl for the measure is not above a grade 8 (?)

Difficulty, discrimination, distractor analysis

Item difficulty analysis — percent of test takers who correctly answer an item

• P = # of examinees answered correctly/# of examinees

Discrimination analysis — On tests of master, to what extent does the item discriminate between test takers who truly differ on the construct

• Compare difference in performance amongst two groups (top test performers vs. bottom test performers)

  • D = Pt - Pb

Distractor Analysis — Evaluating the incorrect options (distractors) in multiple choice questions to determine their effectiveness and learn more abt students

• Examines how often each distractor is chosen by students and identified any patters that might indicate problems w the question or the distractors themselves

• Top students are less likely to select a distractor, whereas bottom students are more likely to select a distractor

Item Response Theory

Based on the idea that the probability of a correct/keyed response to an item is a mathematical function of person and item characteristics (look at the slideshow for this genuinely like im clueless)

(SPSS) Correlation

Read handout

(SPSS) Independent Samples T-Test

Read handout

(SPSS) One-Way Between Subjects ANOVA

Read handout

(SPSS) Cronbach’s Alpha, Spearman Brown

Read handout

(SPSS) Exploratory Factor Analysis

Read handout