Lecture 3 psychometrics

What is the 1st step in test construction

Identify the purpose of the test: What am i measuring and why

Why is identifying the purpose important

The purpose determines:

• content

• format

• difficulty

• target population

• interpretation of scores

What is content analysis?

The process of identifying behaviors that represent a construct

Sources used to identify construct behaviors?

1. research literature

2. critical incidents

3. direct observation

4. expert judgement

What is a critical incident

an important real world even that provides information about behavior

What are test specifications

The blueprint of the test

• subdomains

• number of items

• weight of each area

Aptitude/achievemen test vs personality tests

Aptitude/achievement: Correct/incorrect

Personality/attitude: Likert scales, agree/disagree

Guidelines for writing items

1. present tense

2. simple vocabulary

3. short items

4. avoid “always” and “never”

Why should we avoid words like “always” and “never”

They are usually unrealistic and difficult to endorse accurately

What is checked during item review?

• accuracy

• grammar

• bias

• readability

• construct relevance

• clarity

What is preliminary item piloting?

Administering a test to a small sample before final use

finds weak and confusing items. collects statistics

What is dichotomous scoring

Items scored:

Correct/incorrect

0 / 1

pass / fail

examples of dichotomous tests?

math testts

spelling tests

achievement tests

What does the item mean tell you in dichotomous scoring?

Item difficulty

Item mean = .90 (dichotomous)

Easy item. 90% answered correctly

Item mean = .20 (dichotomous)

Hard item. 20% answered correctly

What does item variance tell you?

How much people differ on the item

Why is very low item variance bad?

The item does not discriminate among examinees

Everyone answers similarly

What is Phi coefficient

Correlation coefficient for dichotomous variables

What is non-dichotomous scoring?

Items with multiple response options.

ex: Likert scales

non-dichotomous measures examples

• depression inventories

• anxiety scales

• personality tests

What should the mean of a good Likert item be?

Near the midpoint of the

Why are midpoint means desireable?

The item discriminates among individuals

What is desireable for variance in personality measures?

High variance. It captures low, moderate, and high trait levels

What is pearson r?

Correlation coefficient for continuous variables (nondichotomous)

What does pearson correlation tell us?

How strongly 2 variables are related

What is variability?

The general concept of score spread - Do people differ?

High variablitiy

High, medium, and low scores

Low varibility

all scored the same

What is variance

Numerical measure of variability

Variability vs variance

Variability = concept of spead

variance = measure of spread

What is covariance?

The degree to which 2 variables move together

Positive covariance

both variables increase together

negative covariance

One increases while other decreases

Covariance vs correlaton

Covariance: raw relationship

Correlation: Standardized relationship

Covariance vs variance

Covariance: 2 variables (Do these variables move together)

Variance: 1 variable (How much does this item vary)

Variance example

“I feel sad” - how spread out are responses

Covariance examples

“I feel sad”. “I feel hopeless”. - do the scores rise together?

What is variance-covariance matrix?

Table showing:

variance of each item

covariance btwn all item pairs

What appears on the diagonal of a variance-covariance matrix?

variances

What appears off the diagonal of a variance-covariance matrix?

covariances

Why is the variance-covarianc matrix useful

Helps determine:

• item relationships

• test variance

• internal consistency

Why is test variance important?

Helps determine optimal test length

Correlation vs internal consistency

Correlation: Relationship btwn 2 variables

(Sadness ←> hopelessness)

Internall consistency: Do ALL items on a test hang together (sadness, hopelessness, fatigue, worthlessness)

6 steps of test construction

1, identify purpose

2. identify bx that represent a construct

3. prepare test specification

4. item construction

5. item review

6. preliminary item piloting