Psychometrics - Procedural

Application of concepts, complex definitions

How could you be 95% sure of someone’s true score on the Article Review prelim?

1. Calculate the mean score (x̄) for the test-taker across all attempts.

2. Compute the standard error of the measure (σM) for the AR prelim.

3. Compute (σM)(1.96).

4. Compute x̄ + 1.96(σM) and x̄ - 1.96(σM).

5. It is 95% likely that their true score is between x̄ ± 1.96(σM).

Your EFA produced 4 factors. What would happen to your Eigenvalues if a 5th factor was produced?

The Eigenvalue of factor 1 would decrease, the other Eigenvalues would increase slightly, and ultimately, all Eigenvalues would tend towards one another in value.

Explain the pros and cons of a 2-factor model versus a 17-factor model.

A 2-factor model is more easily replicable but less specific; it has higher external validity but lower internal validity.

A 17-factor model is more specific but harder to replicate; it has more internal validity but lower external validity.

You want to see if your CFA produced a biased factor structure. How would you do this?

Conduct a confirmatory factor analysis (CFA) on a different sample and compare the two factor structures. If the results significantly differ, this may indicate bias in the original model.

You developed a psychometric test and want to see if it is biased. What would Critical Test Theory suggest you do?

1. Compute the item-total correlation.

2. Compute the item discrimination index.

3. Compare between independent groups.

You want to know how well question 17 on the Article Review prelim can discriminate between the ability levels of two similarly performing students, Patrick and Donna. How would you find this?

1. Find the ability level (ϴ) of both Patrick and Donna.

2. Compute and graph for both students: I(ϴ) = p(1 - p)

3. Assess the curves of each student for slope and ability level.

The Article Review prelim has 40 questions. In general, the score of any given item on the test is strongly correlated (r = 0.8) with scores on the rest of the items on the test. How could you compute the homogeneity of the test items?

Compute Cronbach’s alpha, where k = number of test items, m_r = mean inter-item correlation:

⍺ = [km_r] / [1 + m_r(k - 1)]

⍺ = [40×0.8] / [1 + 0.8(40 - 1)]

⍺ = 32 / [1 + 0.8(39)]

⍺ = 32 / 32.2

⍺ = 0.993

Describe the mathematical relationship between reliability and validity.

A test is no more valid than it is reliable. Reliability (r_xxr_yy) is a prerequisite for validity (r).

r ≤ √(r_xxr_yy)

r² ≤ r_xxr_yy

Describe a CFA model diagram.

Latent variables are represented by circles.

Measured variables are represented by squares.

Residual variables (error) are represented by their names.

Arrows are used to describe relationships.

Variables are represented as being vertically aligned.

Residuals are represented on the right, latent variables on the left, measured variables in the middle.

Describe how test length affects reliability.

Increased test length generally enhances reliability, but at a certain point, reliability will harder to estimate through Cronbach’s alpha and better estimated by the Spearman-Brown formula.

You want to assess the extent to which the Article Review prelim assesses for the same knowledge (content) across groups. How would you do this?

1. Pull multiple, independent groups of Ph.D. students (e.g. G1, G2, G3).

2. Conduct an exploratory factor analysis on the prelim for each cohort.

3. Assess for configural, metric, scalar, and strict invariance across groups.

You are looking at a CFA table and notice to two best models have the same ΔX², CFI, TLI, RMSEA, AIC, and BIC. How do you determine which one is the best-fit model?

The model with the lower X² is considered superior. If X² is not significantly different between the two models, the model with fewer factors is considered superior (Occam’s Razor).

During the Article Review prelim, your fellow Ph.D. student leans over and asks you, “hey, what kind of AFI am I looking for in this CFA on question 9?” What should you tell them?

“You really want an alternative fit index below 2.00, but it is acceptable at values of 4.00, or even 5.00. Also, you’re cheating.”

You are given all six factor loadings for a variable in an exploratory factor analysis, and you want to know how much variance in this variable is not accounted for by any other variables. How would you find this?

You can find the unexplained variance (“uniqueness”) by summating the squared factor loadings and then subtracting this sum from 1.

After an exploratory factor analysis, you want to know what percent of the variance in your data can be explained by your model. How would you find this?

Summate the Eigenvalues of each factor, divide this sum by the number of variables in the model, then multiply this quotient by 100 to convert it from proportion to percent.

In a randomized clinical trial, you want to quantify the extent to which the treatments you administered actually caused any decrease in symptoms. How would you do this?

1. Calculate the pooled variance (or use Welch’s t-test if pre-test and post-test variances are not equal)

2. Calculate the standard error of the difference of means from the pooled variance

3. Subtract the mean post-test score from the mean pre-test score

4. Divide the resulting difference in mean scores by the standard error of the difference

You want to assess the internal consistency of the Article Review prelim, but the assumptions for Cronbach’s alpha have been violated because the exam was poorly designed. What can you do?

1. Conduct an exploratory factor analysis on the AR prelim

2. Summate the Eigenvalues (∑λ)

3. Summate the uniqueness values (∑𝛙)

4. Compute ∑λ / [∑λ + ∑𝛙]

As a psychologist working at Patrick’s school, you test his reading level and find that he reads at an 8th grade level. Patrick is in 5th grade. How would you explain this to his parents (who are not psychologists)?

Patrick’s reading level 3 “grades” above the average of students at his current grade-level, but the term “grades” is not literal. It is a unit of reading proficiency. His reading level does not mean he can literally do 8th grade-level literary work.

Describe the exploratory factor analysis process.

1. From an array of variables, a factor is extrapolated from the ones that are the most statistically correlated with one another.

2. Subsequent factors are extrapolated from the variables left over that are the most correlated with one another.

3. This process continues until the final factor extrapolated has an Eigenvalue less than 1.00.

The entire G1 cohort failed the Article Review prelim the first time, but they all passed the second time. Dr. Miles wants you to figure out how likely this was to not be due to chance. How would you do this?

1. Compute the mean score among G1 students from their first attempt at the AR prelim

2. Compute the mean re-take score among G1 students for the AR prelim

3. Compute the standard error of the measure (SEM) for the AR prelim

4. Compute the difference between mean pre-test scores and mean post-test scores

5. Divide that difference by the standard error of the measure

You failed the Article Review prelim but you are pretty sure the exam was biased. Your fellow G1 colleagues were nice enough to lend you their answers for data analysis, and so did the G2 Ph.D. students. The factor analysis machine broke. How can you test your hunch?

1. Rank the items on the AR prelim in order of difficulty

2. Compute the mean item difficulty for every item. Do this separately for each group

3. Compute a Spearman’s Rho between the rankings of the two groups

4. If Spearman’s Rho is too low, the test is biased and Dr. Henn-Haase owes you a million dollars.

Your grade on the Article Review prelim is a lot lower than you wanted it to be. You want to know, theoretically, how far off the prelim is at assessing your true level of familiarity with psychometrics and research design. How would you go about finding that out?

1. Compute the standard deviation of AR prelim scores

2. Compute the reliability coefficient (r_xx) of the AR prelim

3. Multiply the standard deviation by the square root of 1 - r_xx.

Just when you thought you had completed your exploratory factor analysis, Dr. Miles asks you to rotate your data. What do you expect to happen to your data and why?

1. Factor 1’s Eigenvalue will decrease.

2. All other Eigenvalues will increase.

3. Communality values will not change.

4. The overall factor solution will not change.

5. The sum of the Eigenvalues will not change.

6. The axes are being rotated to better fix the data.

You can only either pass or fail the Article Review prelim. You either have to re-take Dr. Miles’ class or you do not. You want to know how effective the prelim is at only passing students who did well in Dr. Miles’ class, and how effective the prelim is at only failing students who will end up having to re-take his class. How could you do this?

1. Multiply the number of students who passed the prelim by the number of students who passed the class. Call this product (+)².

2. Multiply the number of students who failed the prelim by the number of students who failed the class. Call this product (-)².

3. Count up the number of students who took the prelim (which in this case should be the same number of students who took the class). Call this sum (T) for “total”.

4. To find out how many students who passed Dr. Miles’ class went on to pass the prelim, compute (+)² / T.

5. To find out how many students who failed Dr. Miles’ class went on to fail the prelim, compute (-)² / T.