Psychometrics Midterm II

What are 3 equivalent definitions of THEORETICAL RELIABILITY?

THEORETICAL RELIABILITY IN CLASSICAL TEST THEORY IS THE RATIO OF TRUE SCORE VARIANCE TO OBSERVED SCORE VARIANCE:

1. Proportion of variance in X explained by T, not error

pxx’ = var(t) / var(x)

2. 1 minus proportion of error variance

pxx’ = 1 - var(E) / var (X)

3. Squared correlation between X and T

pxx’ = [corr(T,X)]²

ALL ONLY FOR SIMULATION DATA, AS WE TYPICALLY DO NOT HAVE E AND T SCORES IN DATA COLLECTION

What is the relationship between variance and standard deviation?

VARIANCE is STANDARD DEVIATION SQUARED

What are the 3 properties of E and T scores in CTT that allow us to derive RELIABILITY?

THE THREE PROPERTIES OF E AND T SCORES IN CTT:

1. The mean of error scores in a population are equal to zero

2. The correlation of scores E and T in a population is zero

3. The correlation between scores E obtained from different tests or at different times with the same test, is zero

ERROR IS RANDOM AND THEREFORE UNCORRELATED IN CTT

What is RELIABILITY? What is it’s importance in 2 different contexts?

RELIABILITY is the consistency of scores in a test, under CTT, it is also the extent to which observed scores X reflect true scores T and not E

• the only thing divorcing observed X from true score T is ERROR! therefore more error makes a test less reliable/consistent referring to X and T

In a CLINICAL/EDUCATIONAL SETTING: reliability is used to score, so poor reliability can lead to classification errors

In a SCIENTIFIC SETTING: reliability is important, as the strength of a relationship CORRELATION will be underestimated if there is low RELIABILITY

What is the relationship between VALIDITY and RELIABILITY?

We can have reliability without validity, but we CANNOT have validity without reliability

What is TEST-RETEST RELIABILITY? How does it relate to correlation formulaically?

TEST-RETEST RELIABILITY is the correlation between two PARALLEL tests!!

• parallel tests as those having same true score T, uncorrelated error between tests, but the same variance of error

ie. participants taking the exact same measure at 2 different times

As tests are parallel for test-retest reliability, they are equally reliable, and hence equally correlated

pxx’ = corr(X1, X2) reliability and correlation are the same

What is a PARALLEL TEST? What are its 3 things to remember?

PARALLEL TESTS have the same true score T

uncorrelated measurement errors

but equal variances of measurement error/same reliability

• X1 = T + E1 X2 = T + E2

• varE1 = var E2

• corr(E1, E2) = 0

Can tests be parallel if in a second attempt of a test, everyone’s scores increase by 5?

If in X2, everyone’s score increased by 5, either the TRUE SCORE T increased or error is not averaging to 0/not random → systematic error…

• in this case, test-retest will still estimate reliability and correlation, but the measurement is probably flawed for further use

What is PARALLEL-FORMS RELIABILITY? How is it different than a test-retest reliability?

PARALLEL-FORMS RELIABILITY is the correlation between a test and its PARALLEL FORM (similar test, but not identical as in TEST-RETEST RELIABILITY)

• notation of pxx’ comes from this type of reliability

• less PRACTICE EFFECTS, no MEMORY EFFECTS

Why is TEST-RETEST RELIABILITY also called STABILITY COEFFICIENT? How do we minimize instability/unreliability in research?

RELIABILITY or STABILITY OF SCORES is exemplified when considering that if a measurement differs (is unstable/unreliable at different times) it can appear that the true scores have changed!!

• less stability/reliability if measurement error E increases

• less stability/reliability if true score have changed unsystematically over time

THEREFORE some studies use test-retest multiple times over the course of a study

How do you interpret reliability coefficients?

BASED ON A RELIABILITY COEFFICIENT, we can determine that

• ___% of variance in observed scores is due to TRUE SCORE variance and not error

• 1 - correlation% of differences/variance in scores can be attributed to ERROR not true score variance

In two tests where there are different variances of observed score X, what does this imply?

If there is different variances between observed score X in two tests, this implies different error variance E → making these tests NON-PARALLEL!!!!

Parallel tests require equal observed variances

• different magnitude of error, whereas parallel tests need equal variance of ERROR, no correlation between ERROR, and same true score T

If items on the scale all have the same reliability, what does this imply?

Items having the same reliability, mean that they hold the same amount of TRUE SCORE VARIANCE from item to item!!!

What is the reliability of composite Y compared to individual items’ reliabilities? Does composite Y have a bigger or smaller error E? Does composite Y have a better or worse approximation to true score T?

pyy’ > pxx’

The reliability of a full scale (composite) is greater than any individual item (component)

We reduce a lot of composite error through averaging!!! → with the assumption that after infinite trials X = T anyways, so E = 0

• E bar of the COMPOSITE Y is much smaller than individual component error E

Composite Y has a much better proximity to true value T compared to individual components!

What is INTERNAL CONSISTENCY RELIABILITY? What is its formula? What are its assumptions?

INTERNAL CONSISTENCY RELIABILITY is the reliability of a composite (whole scale) from the reliability of components (individual items) - repeated measurements over trials/across items

Y = 1/k (X1 + X2 + …XK)

Y = T + 1/k (E1 + E2 + … + Ek)

Y = T + Averaged E

What is STANDARDIZED ALPHA? What is its formula? What are its assumptions?

STANDARDIZED ALPHA is a practical reliability estimate to assess the internal consistency reliability/reliability of a COMPOSITE!!

k - number of components (items)

rij - average correlation between components (items) - off diagonals on a matrix

alpha st = krij / 1 + (k - 1) rij

What is the SPEARMAN BROWN FORMULA?

The SPEARMAN BROWN FORMULA can predict the reliability of a composite with k parallel components!!!!

pyy’ = k(pxx’) / 1 + (k-1)(pxx’)

• formula can be used to amount k needed for a certain reliability to be et

• formula can also be used to find reliability, when k is doubled, tripled, etc. given that we have reliability of original pyy’ before changing k!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

What happens to the COMPOSITE when there are more components that go into the composite?

Increasing the number of COMPONENTS in a COMPOSITE, increases reliability → recall the ATTENUTATION PARADOX, why studies use more items to falsely inflate reliability

What are the 3 roles of SPEARMAN BROWN FORMULA?

1. To define STANDARDIZED ALPHA - reliability estimate to estimate internal consistency reliability for a composite made up of parallel components

2. To PREDICT MINIMUM NECESSARY SCALE LENGTH for a desired level of reliability - so long as an estimated correlation between the components is known

3. To DEFINE SPLIT-HALF RELIABILITY - a practical reliability estimate that is half internal consistency, half test-retest in spirit!

What is SPLIT-HALF RELIABILITY? How do you compute it? What are its 3 assumptions?

SPLIT-HALF RELIABILITY is a mix between internal consistency reliability and test-retest reliability!!!

1. Split-half reliability is computed through splitting the test in half (trying to make them similar in content), then correlating the two halves → giving reliability of HALF the test

2. Using Spearman Brown Formula with k = 2 and the correlation between two halves!!!!

It is assumed to not be unique (multiple combinations to split test), assuming parallel halves but not items, and only works for even scales

How do INTERNAL CONSISTENCY and TEST-RETEST RELIABILITY differ in their definitions of the TRUE SCORE?

TEST-RETEST RELIABILITY assumes TRUE SCORE is that which is consistent across time - T as the long run average over time

INTERNAL CONSISTENCY assumes TRUE SCORE as how well items hang with eachother - T as the long run average over items

What are TAU-EQUIVALENT COMPONENTS?

TAU-EQUIVALENT ITEMS are those that have the same variance of true scores T, but different variance of error E

• var(E1) ≠ var (Ek)

Observed scores X have different reliability

Correlations among X’s are not equal - more noise leads to lower correlations

What is P-HACKING?

P-HACKING is one of 3 questionable research practice (QRP) - looking at multiple DVs, Excluding conditions of a study, Excluding outliers

SOLUTION: preregistering, publicly stating a hypothesis a priori