Psychometric Lecture 5

What is an achievement test

Measures acquired knowledge or skills that have already been learned

ex: spelling, EPPP, comprehensive exams

What is an aptitude test

Measures future potential or capacity to learn

ex: SAT, cognitive aptitude test

Achievement vs aptitude test

Achievement: What have you learned

Aptitude: What could you learn/do?

What is Criterion referenced testing

A test that evaluates performance against a fixed standard or cutoff rather than other people

What is a domain score?

The proportion of items answered correctly within a content domain

What is mastery allocation

Classification of individuals into: Master/ nonmaster

What is an absolute threshold?

The cutoff score used to dtermine mastery

ex: 140 = pass 139 = fail

Why are traditional reliability coefficients less useful for criterion-referenced tests

Because the concern is classification accuracy (pass/fail) rather than score consistency

What is absolute error?

Using an examinees score as mean score as a representation of his true universe score

What contributes to absolute error?

• item difficulty

• poorly written questions

• Measurement problems

Lower absolute error means what?

The observed score more accurately reflects true knowledge

Reliability of classification asks…

Can I trust the pass/fail decision

Traditional reliability asks…

Can I trust the score

What do p11 and p00 represent?

Correct classifications.

P11: Master correctly classified

P00: Nonmaster correctly classified

Formula for reliability of classification?

P = p11 + p00

What is a z-score

A standardized score indicating how many standard deviations a score is above or below the mean

Why convert raw scores into z-scores?

Standardization, communication, objectification, comparison across tests

What is a percentile rank?

The percentage of people who scored below a particular score.

Does a percentile rank indicate % correct?

No, it indicates relative standing. Not % correct

Why do psychometricians want greater test variance?

Greater variance allows better disrimination among individuals

What happens when variance is low?

Everyone looks similar

Individuals cannot be distinguished well

What generally happens when items are added?

Test variance usually increases.

Why don’t additional items increase variance forever?

Diminishing returns. Items become redundant and contribute little new information

What is literal meaning?

The actual semantic meaning of the questions

What is pragmatic meaning?

The meaning participants infer beyond the words

Self report responses are influenced by…

1. wording

2. context

3. response options

4. previous questions

5. reference periods

What happens with short reference periods?

respondents focus on frequency (last week, yesterday)

What happens with long reference periods?

Respondents focus on intensity and memorable events (last year)

Why can response scales change answers?

people use the scale itself as information when answering

Scaling/response format effects

When identical bxs produce different answers because respondenst options changed

Why are preceding question effects problematic

Responses may reflect context instead of the construct

Advantages of internet testing

• cheaper

• fasster

• easier updates

• more respondents

• easier scoring

• adaptive testing

What is adaptive testing

Questions change according to the individual’s ability level

What is the major informed consent concern online?

Many click “agree” without reading or understanding the information

Why is reliability and validity a concern online?

Many tests were validated in paper and pencil formats, not online

Why are norms a concern for international testing?

Norms established in one population may not apply to another

What confidentiality concerns exist in online testing?

• data breaches

• security issues

• data mining

• unauthorized access

Why should psychological tests not be used by unqualified individuals?

Interpretation requires professional training, and misuse can lead to incorrect conclusions

Why must scores be interpreted in context?

Scores can be affected by:

• ESL status

• culture

• religion

• environment

• testing conditions

What is unstandardized administration

Individuals complete a test under different conditions

This threatens consistency and interpretation

Criterion referenced =

did the person meet the standard ?

Reliability of classification

Can i trust the pass/fail decision

how is absolute error calculated

by summing al the error variance (associated w variable item difficulty & errors of measurement)

what does achievement test give you?

• relative position (z score)

• degree to which the person has attained the goal