PTPR 1: Psychological Measurement

levels of psychological measurement

• psychological construct:

  • unobservable

  • theoretical concept of psychological differences between individuals

• psychological test:

  • observable

  • measurement instrument to quantify individual differences in the construct

how psychological tests differ

• content: depends on construct
• response required: open-ended or closed-ended
• method of administration: individual vs. group
• use: 
• criterion-referenced: determined whether someone passes a cutoff (e.g. uni exams, driving test)
• norm-referenced: compare score to population (e.g. IQ test)
• timing:
• speeded test: time limited, relatively easy but many questions, see how far one comes
• power test: not time limited, different difficulty, see what one can solve
• meaning of indicators: formative vs. reflective
• reflective (effect) indicators: intelligence, personality test - responses are caused by the construct
• formative (causal) indicators: SES - the indicators are what define construct 

reflective measurement

"= most used in Psychology, assumed that construct causes differences in test scores.

• item responses are indicators of construct
• items are necessarily correlated (higher value on construct -> higher scores)

"

formative measurement

"= item responses (indicators) define construct

• items are not necessarily correlated with each other, independently contribute to construct

"

challenges in psychological measurement

• complexity (!): psychological phenomena are complex, high-dimensional concepts; need to carefully isolate a psych. dimension (construct) and measure it, can't really define people using numbers.
• reactivity: people respond differently knowing they are being measured - demand characteristics, social desirability, malingering
• observer bias: expectation of researcher may affect the test
• composite scores: multiple item scores need to be combined into one score
• sensitivity: unknown beforehand how sensitive a scale should be
• too few response categories, miss out individual differences
• too many response categories, participants can't meaningfully distinguish different categories
• awareness: test administrators don't know about psychometric quality of the test

dimensionality

"= number of constructs that a test measures

• unidimensional test: measures a single construct (e.g. D2 test of attention, primary school arithmetic test) - conceptual homogeneity
• multidimensional test: measures multiple constructs (e.g. Big Five, WAIS-IV intelligence test) - in psychology most constructs are correlated

"

dimensionality analysis

factor analysis: a statistical method to study the dimensionality of a test

• Exploratory factor analysis (EFA): no theory about factor structure.
• Confirmatory factor analysis (CFA): clear theory about factor structure.

EFA steps

"

1. Correlation matrix
2. Check Eigenvalues
3. Select no. of factors
4. Interpret factors

"

EFA correlation matrix

"summarises all possible correlations among items.

two clusters (programming, biology - exact; english, spanish - language) -> indicates two dimensions in data"

Eigenvalues

"EFA assumes no. of dimensions = no. of items.
Then see which dimensions are more important.
Eigenvalue - shows how much variability the factor can account for in the data.

Decreasing order, Factor 1 is most important. See how many of these factors cumulatively account for most of the data to see which are most important."

Select a number of factors

"
1. Kaiser criterion: select all factors with Eigenvalue >1 (big nono), doesn't perform very well.
2. Scree plot:

see which factors aren't important enough, those before the inflection point (so 1 and 2)
inflection point: where Eigenvalues stop decreasing greatly"

Interpreting factors

"factor loading: correlation between item and factor - range from -1 to 1

to make it easier to read (values don't differ so greatly), orthogonal rotation: put items on axis

then rotate it:

Varimax - two dimensions are uncorrelated, 90 degree angle
if 90 degree angle is relaxed (oblique rotation), might come up with a better explanation - now they're correlated :D
"

factor scores

calculate the scores people get on the important dimensions, use all x scores as measures for the construct

• reduce data (use the factor scores in a t-test)
• has advantages over the sum score (composit score)
• more natural measure of construct
• takes dimensionality into account
• avoid multicollinearity: too many values will be highly correlated, but few variables will be less correlated.

pattern matrix vs structure matrix

"pattern matrix:
factor loadings controlled for the correlation between the factors (best interpretable) - you look if there is an oblique rotation 

rewrite/take out red items
analyse items, interpret which dimensions they correspond to.
structure matrix:
correlation between item and factors (more difficult to interpret)
factor matrix:
factor loadings BEFORE ROTATION
"

contraindicative item

higher scores on the item indicate lower levels of the construct
score is reversed when calculating (in 1-5 scale, 2 becomes 4)

validity

whether the inference from an observable behaviour to an unobservable psychological attribute is reasonable.

hypothetical constructs

or latent variables, theroretical psychological characteristics/processes that cannot be directly observed (intelligence, self-esteem, memory etc.)

psychological test

systematic procedure for comparing the behaviour of 2 or more people.

1. tests involve behavioural samples of some kind
2. behavioural samples must be collected in some systematic way
3. purpose of the tests is to detect differences between people - interindividual (or different points in time/situations - intraindividual).

psychometrics

= science concerned with evaluating attributes of psychological test:

1. type of information (usually scores) generated by the use of tests
2. reliability of data
3. issues concerning the validity of data

These attributes are also unobservable, so must be estimated.

the validation process

process by which psychologist accumulates evidence that there is an association between individuals' scores on the test and the true level of the construct.

differential psychology

= the study of individual differences - intelligence, aptitude, personality

• contrasts with experimental psychology, focused on the average person

variability

= the degree of differences within a set of test scores/among a psychological attribute - dehree to which scores in a distribution deviate from the mean.

• interindividual variability - differences between people
• intraindividual variability - differences in one person over time/under different circumstances

• most common statistical values: variance and standard deviation

consistency/covariability

= the degree to which variability in one set of test scores corresponds with (is consistent with) variability in another set of scores

conceptual issues and statistical indexes

""

central tendency

"= ""typical"", most representative score in the distribution

• can be reflected by multiple statistical values, most often the mean.

"

variance

"
mean of squared deviations of each score.

• numerator = sum of squares
• st. dev = s (square root of variance)

Size of variance is determined by:

• degree to which scores in the distribution differ from each other;
• metric of the scores.

"

four factors to consider when interpreting variance/st.dev.

• can't be <0
• can't simply interpret what a big/small one is
• most interpretable when put in context (comparing distribution scores based on the same measure)
• importance lies in effects to other values that are more directly interpretable

association between two varibales

two types of info required:

• direction of the association
• magnitude of the association

covariance calculation

"

covariance provides information about direction of the association: positive, direct association, negative, inverse association.

1. compute deviation of each score from mean
2. compute ""cross-products"": multiply individual's two deviation scores
3. compute mean of cross-products

Two factors affect size of covariance:

• strength of association
• metrics!

-> limited interpretability

"

correlation coefficient

"represents linear association, says something about magnitude and direction.
"

variance of composite scores

"= variability of each item within composite, correlations among items

• total test score variance will depend solely on item variability and the correlation between item pairs

"

variance in binary items

s²=p*q
depends only on the proportion of each of the 2 possible responses

interpreting test results

"two facets to meaning of test scores:

• basic quantitative meaning - relatively high/low
• psychological implications of test scores - what does it actually mean

"

z score

"= standard scores


benefits:

• express test scores unambiguously
• can be used to compare scores across different unit tests
• express scores in relative terms

"

converted standard scores

= standardised scores, converted into values that are easier to understand

normalised scores

scoring mechanism that produces scores that are normally distributed

• there might be a discrepancy between our theory-based belief and the actual test data

one way in which test developers have tried to solve this nonnormality problem is by transforming the imperfect distribution into a distribution that more closely approximates a normal distribution - normalization transformations or area transformations.

two assumptions: 

• levels of the psychological attribute are normally distributed
• the actual test data obtained in sample are imperfect reflections of the distribution of the construct itself

test norms

• test users can use reference sample scores as a frame of reference for interpreting other scores

In research, might not be reliable:

• norms might not be available for certain tests
• researchers not usually interested in interpreting individual scores

3 questions about dimensionality

1. how many dimensions are reflected in the test?
2. are the dimensions correlated with each other?
3. what are the dimensions?