College 3

Test theory

validity

does the test measure what it is supposed to measure?

aim of factor analysis

to cluster item into less variables (factors), while retaining as much information as possible

factor score

weighted sum of item scores of subtest scores

exploratory factor analysis

what is the structure of the test?

confirmation factor analysis

can you confirm the assumed structure of the test (so there is already a structure, can you distinguish those subscales?)

component analysis

• PCA: principal component analysis (exploratory)

• MGM: multiple group method (confirmatory)

stappelplan component analysis

1. determine the weights

   1. MGM: chosen by researcher (0/1)

   2. PCA : optimal estimation on the basis of observed data

2. correlations of all variables (=items) on all factor scores

   1. we call these correlations for some models: loadings on a factor

3. interpretations

   1. variables with a high correlation on the same factor measure similar content

   2. label the factor

4. proportion explained variance

   1. how well do the factors represent the variation in the data?

   2. variance accounted for (VAF), usually between .30 - .80

   3. more factors = higher VAF

variance accounted for (VAF)

how well does the model fit the data, how well does my factor score describe my data

ortogonal

uncorrelated factors, constructs are independent

oblique

correlated factors (depended constructs)

MGM

• is the expected grouping of variables found in the data? Weights 1 or 0

• correlation between variables and proposed factors

• expected: for factor q the variables with weight 1 correlate higher than variables with weight 0

PCA

find an x number of factors that explain as much variance as possible

• find ‘optimal’ weight for these variables

• PCA tries to come up with a weighted combination of the item scores that capture most variance

• factors are found one by one

finding factors PCA

• find weights so that the factor explains maximum variance

• result: first principal componant (PC)

• then, for all variables, the residuals of the variables are computed by subtracting that part of the variables that is explained by the first PC

• find weights from the residuals that explains maximum variance

• result: second principal component

  • First PC explains most variance, then the second etc

  • all PC’s are uncorrelated

rotations

can make factor-item correlation easier to interpret. Aim is to substitute PC’s by new factors with in total the same amount of VAF

How do we determine the numver of PC’s?

• researchers may have an idea about the number of factors, procedure stops when this number of factors is reached

• On the basis of criteria

  • Kaiser’s criterion

  • Scree scriterion

Kaiser’s criterion

• eigenvalue > 1

• Eigenvalue is related to the amount of explained variance associated with the component

• This criterion often overestimates the number of PC’s

Scree criterion

• as much VAF as possible with the least amount of factors

• number of factors before ‘bend’ in screeplot

Item response theory (IRT)

• say something about the quality of your test

• central in IRT is item characteristic curve (ICC)

  • can be used to investigate quality of items

  • gives the relation between a trait value and the probability of answering an item correctly

  • curve can be described by parameter models

parameter models

• 1 parameter: only the difficulty (kan je in grafiek zien omdat de lijnen elkaar niet kruisen

• 2 parameter: difficulty and slopen

• 3 parameter: difficulty, slope and guassing (kan je zien als de grafieken niet allemaal bij 0 beginnen op de x-as)

guessing parameter (c_i)

lower asymptote, the probability of giving a correct answer if you don’t know anything

Item difficulty (b_i)

the point on the trait scale where the probability of giving a correct answer equals (1+ c_i)/2

Item discrimination (a_i)

the slopen of the ICC, the higher the slopen, the better and item discriminates between persons with different trait scores