Chapter 8 - Textbook

test development

an umbrella term for all that goes into the process of creating a test

test conceptualization *

idea for a test is conceived

test construction

a stage in the process of test development that entails writing test items (or re-writing or revising existing items)

test tryout

a test  is administered to a representative sample of testtakers under conditions that simulate the conditions under which the final version of the test will be administered

test revision

refers to action taken to modify a test’s content or format for the purpose of improving the test’s effectiveness as a tool of measurement

LGTBQIA2S+

Lesbian, Gay, Bisexual, Transgender, Queer and/or Questioning, Intersex, Asexual, Two-Spirit, plus innumerable other affirmative ways to self-identify

asexuality

• a sexual orientation characterized by a long-term lack of interest in a sexual relationship with anyone or anything

human asexuality

as an absence of sexual attraction to anyone at all

pilot work

preliminary research surrounding the creation of a prototype of the test

Scaling

the process of setting rules for assigning numbers in measurement

rating scale

a grouping of words, statements, or symbols on which judgments of the strength of a particular trait, attitude, or emotion are indicated by the testtaker

summative scale

final test score is obtained by summing the ratings across all the items

Likert scale*

method of paired comparisons

• Testtakers are presented with pairs of stimuli (two photographs, two objects,

  two statements), which they are asked to compare. They must select one of the stimuli according to some rule; for example, the rule that they agree more with one statement than the other, or the rule that they find one stimulus more appealing than the other

comparative scaling *

categorical scaling

Stimuli are placed into one of two or more alternative categories that differ quantitatively with respect to some continuum

Guttman scale *

scaling method that yields ordinal-level measures

scalogram analysis

an item-analysis procedure and approach to test development that involves a graphic mapping of a testtaker’s responses

item pool

the reservoir or well from which items will or will not be drawn for the final version of the test

item format

Variables such as the form, plan, structure, arrangement, and layout of individual test items

selected-response format *

require testtakers to select a response from a set of alternative responses

constructed-response format

require testtakers to supply or to create the correct answer, not merely to select it

multiple-choice format

three elements: (1) a stem, (2) a correct alternative or option, and (3) several incorrect alternatives or options variously referred to as distractors or foils.

binary-choice item

multiple-choice item that contains only two possible responses

true–false item

type of selected-response item usually takes the form of a sentence that requires the testtaker to indicate whether the statement is or is not a fact

completion item"*

short-answer item *

essay item

a test item that requires the testtaker to respond to a question by writing a composition, typically one that demonstrates recall of facts, understanding, analysis, and/or interpretation

item bank

a relatively large and easily accessible collection of test questions

computerized adaptive testing (CAT)

an interactive, computer administered test-taking process wherein items presented to the testtaker are based in part on the testtaker’s performance on previous items

floor effect

the diminished utility of an assessment tool for distinguishing testtakers at the low end of the ability, trait, or other attribute being measured

ceiling effect

the diminished utility of an assessment tool for distinguishing testtakers at the high end of the ability, trait, or other attribute being measured

item branching

ability of the computer to tailor the content and order of presentation of test items on the basis of responses to previous items

class scoring

testtaker responses earn credit toward placement in a particular class or category with other testtakers whose pattern of responses is presumably similar in some way

category scoring

testtaker responses earn credit toward placement in a particular class or category with other testtakers whose pattern of responses is presumably similar in some way

ipsative scoring

comparing a testtaker’s score on one scale within a test to another scale within that same test

Pseudobulbar affect (PBA)

a neurological disorder characterized by frequent and involuntary outbursts of laughing or crying that may or may not be appropriate to the situation

item analysis

different types of statistical scrutiny that the test data can potentially undergo at this poin

item-difficulty index*

item-endorsement index*

item-validity index

a statistic designed to provide an indication of the degree to which a test is measuring what it purports to measure

item-discrimination index

symbolized by a lowercase italic “d” (d)

compares performance on a particular item with performance in the upper and lower regions of a distribution of continuous test scores

item-characteristic curve

a graphic representation of item difficulty and discrimination

guessing*

item fairness

refers to the degree, if any, a test item is biased

biased test item

an item that favors one particular group of examinees in relation to another when differences in group ability are controlled

qualitative methods

are techniques of data generation and analysis that rely primarily on verbal rather than mathematical or statistical procedures

Qualitative item analysis

various nonstatistical procedures designed to explore how individual test items work

“think aloud” test administration

qualitative research tool designed to shed light on the testtaker’s thought processes during the administration of a test

expert panels* 

sensitivity review *

a study of test items, typically conducted during the test development process, in which items are examined for fairness to all prospective testtakers and for the presence of offensive language, stereotypes, or situations

cross-validation

the revalidation of a test on a sample of testtakers other than those on whom test performance was originally found to be a valid predictor of some criterion

validity shrinkage

decrease in item validities that inevitably occurs after cross validation of findings

co-validation

test validation process conducted on two or more tests using the same sample of testtakers

co-norming*

anchor protocol

a test protocol scored by a highly authoritative scorer that is designed as a model for scoring and a mechanism for resolving scoring discrepancies

scoring drift

discrepancy between scoring in an anchor protocol and the scoring of another protocol

differential item functioning (DIF)

an item functions differently in one group of testtakers as compared to another group of testtakers known to have the same (or similar) level of the underlying trait

DIF analysis

test developers scrutinize group-by-group item response curves, looking for what are termed DIF items.

DIF items

items that respondents from different groups at the same level of the underlying trait have different probabilities of endorsing as a function of their group membership