2.2 Test Tryout & Item Analysis

Item Analysis

Stage of test development and statistical procedure to analyze items and evaluate test items

Item

Suggests a sample of behavior of an individual

Table of Specification

A blueprint of the test in terms of number of items per difficulty, topic importance, or taxonomy

Item-Difficulty Index

Index obtained by calculating the proportion of the total number of test takers who answered correctly. 0-1 The higher the value, the easier the question

Item Endorsement Index

Statistics that provides, not a measure of of the percent of people passing the item but a measure of the percent of people who said yes to, agreed with, or otherwise endorsed the item

P = R/T

Formula for the Item-Difficulty Index

Very Difficult

Level of Difficulty of 0.00-0.19

Difficult

Level of Difficulty of 0.20 - 0.39

Average

Level of Difficulty of 0.40 - 0.60

Easy

Level of Difficulty of 0.61 - 0.79

Very Easy

Level of Difficulty of 0.8 - 1.00

Optimal Level of Item Difficulty (Optimal Difficulty)

Refers to the ideal level of difficulty for the questions on a test or assessment, designed to maximize the effectiveness and quality of the test

P = 1+c/2

Formula for the Optimal Level of Difficulty

Should be roughly midway between the chance score and a perfect score

General Principle of Item-Difficulty

Chance Score

The proportion of correct answers expected if test-takers were simply guessing

1/4 , 1/5

What is the chance score for a 4-option and 5-option multiple choice?

Perfect Score

1.0 (or 100% correct

Slightly higher than midway between the chance score and 1.0

What is the ideal difficulty for a multiple choice item?

Item-Reliability Index

Index that indicates the internal consistency of a test

May have limited applicability

Internal consistency of an item reliability index of 0.69 & below

Adequate

Internal consistency of an item reliability index of 0.70-0.79

Good

Internal consistency of an item reliability index of 0.80-0.89

Excellent

Internal consistency of an item reliability index of 0.90 & above

Item-Validity Index

Index designed to provide an indication of the degree to which a test is measuring what it purports to measure. The greater it is, the greater the criterion-related validity

Unlikely to be useful

Interpretation of a Validity Coefficient of Below 0.11

Varies on the circumstances

Interpretation of a Validity Coefficient of 0.11-0.20

Likely to be useful

Interpretation of a Validity Coefficient of 0.21-0.35

Very beneficial

Interpretation of a Validity Coefficient of above 0.35

Item-Discrimination Index

Index that indicates how adequately an item separates (discriminate) between high scores and low scores on an entire test

D = Rh - Rl /N

Formula for the Item-Discrimination Index

Poor item

Verbal Description of an Item-Discrimination Index of 0.09-0.019

Fair item

Verbal Description of an Item-Discrimination Index of 0.20-0.29

Good item

Verbal Description of an Item-Discrimination Index of 0.30-0.39

Very Good item

Verbal Description of an Item-Discrimination Index of 0.40 & above

Item Characteristic Curves

Graphic representation of item difficulty and discrimination. The steeper the shape, the greater the item discrimination

Item Fairness

The degree, if any, a test item is biased

Biased Test Items

An item that favors one particular group of examiners in relation to another when differences in group ability are controlled

Speed Test

Item analyses of tests taken under speed conditions yield misleading or uninterpretable results. The closer an item is to end of test, the more difficult it may appear to be

Qualitative Item Analysis

General term for various nonstatistical procedures designed to explore how individual test items work

“Think Aloud” Test Administration

Designed to shed light on the test taker's thought processes during the administration of test

Extra Panels

Panel members met with the publisher to obtain an understanding of the history and philosophy of test battery and to discuss and define the problem of bias