B. 4. Test Conceptualization & Test Construction

Test conceptualization

Test construction

Test tryout

Item Analysis

Item Revision

Stages of Test Development

Test Conceptualization

First stage of test development that involves the process of conceptualizing the construct, items included, norm-referenced/criterion referenced, pilot study, and overall design

Test Construction

Second stage of test development that entails writing test items (or rewriting or revising existing test items), as well as formatting items, setting scoring rules, and otherwise designing and building a test

Item Pool

The reservoir where items will be drawn from for the final version of the test. It is generally advised to have twice the number of items that the final version of the test will contain

Item Banks

These are relatively large and easily accessible collection of test questions

Computerized Adaptive Testing (CAT)

Refers to an interactive, computer administered test-taking process where items presented to test takers are based in the part on the test taker’s performance on the previous items

Floor Effect

Low end of ability, trait, or other measurable attribute

Ceiling Effect

High end of ability, trait, or other measurable attribute

Item Branching

The ability of the computer to tailor the content and order of presentation of items on the basis of responses to previous items

Item Format

It is the form, plan, structure, arrangement, and layout of individual test items

Dichotomous format

Format where each item has 2 choices

Polytomous format

Format where each items has more than 2 choices

Category Format

A format where respondents are asked to rate a construct

Checklist

A format where the subject receives a long list of adjectives and indicate whether each one is characteristic of himself/herself

Guttman Scale

A format where answers are arranged sequentially from weaker to stronger expressions of attitudes, beliefs, or feelings being measured

Selected-Response Format

Format where test takers select a response from a set of alternatives

Multiple choice, matching item, binary choice

Types of selected-response format

Stem

Correct answer

Incorrect alternatives such as distractors or foils

Three elements of multiple choice formats

Effective Distractors

Alternative answers chosen by both high- and low-performing groups, which enhances the consistency of a test result

Ineffective Distractors

Alternative answers that may hurt the reliability of the test because they are time-consuming to read and can limit the number of good items

Cute Distractors

Alternative answers that are less likely to be chosen, may affect the reliability of the test takers, who may guess from the remaining options

Constructed-Response Format

Format where test takers supply or create the correct answer

Scaling

Process of setting rules for assigning numbers in measurement

Notion of Absolute Scaling

Procedure that involves obtaining a measure of item difficulty across samples of test takers who vary in ability

Paired Comparison

Rank Order

Constant Sum

Q-Sort Technique

Types of Comparative Scaling

Paired Comparison

Type of comparative scaling that produces ordinal data by presenting pairs of two stimuli to compare

Rank Order

Type of comparative scaling where respondents are presented with several items simultaneously and are asked to rank in order or priority

Constant Sum

Type of comparative scaling where respondents are asked to allocate a constant sum of units, such as points, among set of stimulus objects with respect to some criterion

Q-Sort Technique

Type of comparative scaling where respondents are asked to sort objects based on similarity with respect to some criterion

Continuous Rating

Itemized Rating

Likert Scale

Visual Analogue Scale

Semantic Differential Scale

Stapel Scale

Summative Scale

Thurstone Scale

Ipsative Scale

Types of Non-Comparative Scaling

Continuous Rating

Type of non-comparative scale where respondents are asked to rate objects by placing a mark at the appropriate position on a continuous line that runs from one extreme of the criterion variable to the other

Itemized Rating

Type of non-comparative scale that has numbers or brief descriptions associated with each category

Likert Scale

Type of non-comparative scale where respondents indicate how strongly they agree or disagree with carefully worded statements that range from very positive to very negative

Visual Analogue Scale

Type of non-comparative scale that has a 100-mm line that allows subjects to express the magnitude of an experience or belief

Semantic Differential Scale

Type of non-comparative scale that measures an individual’s attitudes or emotional reactions to a concept or object by using a series of bipolar adjectives or phrases

Stapel Scale

Type of non-comparative scale that uses a numerical scale, typically presented vertically, with a single adjective in the middle to measure respondents' attitudes or opinions about a specific subject

Thurstone Scale

Type of non-comparative scale that uses a set of statements about a topic, with each statement assigned a numerical value reflecting the respondent’s attitude towards it

Ipsative Scale

Type of non-comparative scale where respondents distribute a fixed number of points across different attributes, highlighting their relative strengths and weaknesses within themselves

Class Scoring (Category Scoring)

Cumulative Model

Ipsative Scoring

Ways of Scoring Items

Class Scoring (Category Scoring)

Scoring where test taker responses earn credit toward placement in a particular class or category with other test takers whose pattern or responses is presumably similar in one way

Cumulative Model

The higher the score on the test, the higher the test taker is on the ability, trait, or other characteristic that the test purports to measure

Ipsative Scoring

Comparing a test taker’s score on one scale with a test to another scale within that same test