PSYC 406: Lecture 10

Question, construct, item, response, score, items, quality, items

From test user to test builder

• a good score starts with a ___ that works

1. ___: what do we want to measure ?

2. ___: what question captures it ?

3. ___: how will people answer it ?

4. ___: can we interpret the result ?

Need well designed ___. If the item ___ is weak, problems can arise before we even look at scores.

A psychological test is only as strong as the ___ that generate its scores.

Where, understand, retrieve, judge, report, judgement, understand, remember, judge, respond

If we know ___ answers can go wrong, we can design better items

1. ___: what is the question asking ?

2. ___: what info comes to mind ?

3. ___: how do i turn memory into an answer ?

4. ___: which option best fits ? Map ___ into one of the response options.

Good items help respondents ____, ___, ___ and ___ accurately

Several, comprehension, ambiguity, retrieval, judgement, worst, response, interpreted, Tourangeau, 7, academic

Response process: one bad item can fail at ___ stages

Example item: How often are you stressed ?

1. ___: what counts as stressed? Academic? Social? Financial ?

• There is some ___ at the very start; some challenge

2. ___: what period? Today ? This week? This semester ?

3. ___: should i average ? Remember ___ moments ?

• how do i combine info from memory into an answer ?

4. ___: what does often mean? Compared with whom ?

• some words can be ___ differently by people.

The people who created these guidelines are ___ et al. (2000)

Better item: “During the past ___ days, how often did ___ workload make you feel tense or overwhelmed ?”

Multiple, exercise, time frame

Diagnose the item before fixing it

• for each item, identify the likely response-process problem

A. Do you want to be rich and famous ? Here, ___ ideas at once

B. In the last week, did you exercise ? What counts as ____? More details needed/

C. How often are you sad ? Without a ___ ___, hard to provide an answer.

Grasping, what, scope, what, boundary, period, key terms, instructions, respondents, ambiguity, multiple, one thing, 2, whom, what, when, clarity, respondents, interpret, answering

Comprehension: ___ the question

• before answering, respondents must know exactly ____ the item is asking

1. Notice the question and ___:

• ____ is the item about, and what is inside the ___?

• Example: “in the last 6 months” tells respondents which ___ counts

2. Understand ____ ___:

• terms and ___ should mean the same thing to different ___

• Example: “Fair” can mean just, reasonable or attractive

3. Avoid ___ or ___ questions

• one item should ask ___ ___

• Example: Do you want to be rich and famous ? Mixes ___ ideas

Weak item: Do you feel supported ? Problem: supported by ___, for ___, and ___?

Stronger item: “during the past month, ow supported did you feel by your close friends when dealing with school related stress?

___ means the same question is understood in the same way by different ___. If respondents ___ the item differently, they are not reall ___ the same question

Memory, facts, episodes, general impressions, memory, specific, general impression, forget, accessibility, recent, distinctive, faster, selective, unusual, anchors, dates, events, lists, timing, cue, guess, longer, vaguer, estimates, shortcuts, heuristics

Retrieval: bringing information to ___

• respondents search memory for ___, ___ or ___ ___

1. Search ___

• people may recall ___ events or rely on a ___ ___

• Example: routine short doctor visits are easy to ___

2. ___ matters

• ___ or ___ events are easier to retrieve

• Example: one major stressful week may come to mind ___ than ordinary weeks

• Memory is ___: ___ items come to mind more easily

3. Use helpful ___

• ___, ___, ___ can support recall

• Specify ___: during the past 7 days, since the start of the semester.

Good items ___ the right memories instead of forcing people to ___. The ___ or ___ the recall task, the more respondents rely on ___ and ___ (aka ___) .

• dont want to lead respondents, just want memory aids so they dont rely on heuristics

Answer, incomplete, combine, infer, estimate, combine, average, shortcuts, availability, extrapolation, rounding, attitudes, preformed, spot, weak, wording, clear, shortcuts

Judgement: forming an ___

• when memory is ___, respondents ___< ___ or ___

1. ___ info

• respondents may sum or ___ fragments from memory

• Example: estimates yearly visits from a typical month

2. Use ____

• ___ , ___, and ___ can shape answers

• Example: reporting 30 days when exact recall is hard. OR “this week was busy, so the whole semester must have been busy”

3. Construct ___

• some opinions are ___; others are built on the ___ from what comes to mind

• Example: a ___ attitude can be influenced by ___.

A good item makes the judgement task ___ enough that ___ do not dominate the answer.

Selecting, giving, internal, options, categories, fit, fuzzy, acceptability, acceptable, judgement, social desirability, sacrifice, motivation, ability, acceptable, plausible, never, sometimes, often, boundaries, boundaries, measurement, formatting

Responding: ___ and ___ a response

• respondents translate their ___ answers into the available response ___

1. Map to ___

• the answer has to ___ one of the options

• Example: the line between “agree” and “strongly agree” may be ___

2. Edit for ___

• people may adjust their answers to look ___ or avoid ___. ___ ___ effect.

• Example: underreporting drug use in an interview

3. ___

• when ___ or ___ is low, respondents may pick the first ___ option

• Example: choosing the first ___ answer in a long survey

Bad categories: ___/___/___. Problem: category ___ are vague

Better categories: 0 days/ 1-2 days/ 3-4 days/ 5-7 days. ___ are clearer.

Response options are part of ___, not just ___

One, item, instructions, key terms, reference, forgetting, missing, memory cues, routine, heuristic, timing, rounding, typical, social desirability, categories, sacrificing, safer, where

Common failure modes

• response errors often map onto ___ of the four stages

1. Comprehension

• Misunderstanding the ___, ___, ___ ___, or ___ period

• Example: “exercise” means different things to different people

2. Retrieval

• ___, ___ info, or using poor ___ ___

• Example: ___ events are undercounted

3. Judgement

• ___-driven estimation, ___ errors, or ___

• Example: a recent event feels ___

4. Response

• ___ ____ effect, unclear ___, or ___

• Example: choosing a ___ answer

When an item fails, ask: ___ in the response process did the failure happen ?

Design, dessin, diagnose, improve, construct fit, comprehension, understand, retrieval, mind, judgement, response, supports

Item-writing framework: from response process to item design

• we now know where responses fail; let’s turn that into ___ rules. Use the same response stages to ___, ___ and ___ items

Before the response process: ___ ___: does the item represent the intended […] or facet ?

1. ___: can respondents ___ exactly what is being asked?

2. ___: can they bring the relevant info to ___?

3. ___: can they form the […] you are asking them to make?

4. ___: can they express that jdugement using the response options ?

A good item ___ the respondent at every stage of answering.

Easy, interpret, focused, one, double barrel, clear, direct, concise, double negatives, jargon, complexity, scope, key, meaning, reference, interpret, same

Comprehension: make the item ___ to understand

• the respondent should ___ the item that way the developer intended

1. Keep each item ___

• ask ___ thing at a time. No ___ ___ questions

• Poor: I feel stressed and unable to manage my workload

• Better: i feel stressed by my academic workload

2. Use ___, ___ language

• keep wording ___; avoid ___ ___, ___ and unnecessary ___

• Poor: i am not usually unable to cope with deadlines

• Better: i can cope with academic deadlines

3. Define ___ and ___ terms

• make the intended ___ and ___ period clear

• Vague: how often are you stressed

• Clearer: during the past 7 days, how often did you feel stressed about university work ?

If respondents ___ the items differently, they are not really answering the ___ question

Recall, recall able, recent, distinctive, routine, distant, reference periods, when, memory cues, dates, events, examples, lists, wording, recall

Retrieval: help respondents bring the right info to mind

• good items ask for info people can reasonably ___

1. Ask about ___ info

• ___ or ___ experiences are easier to remember than ___ or ___ events

2. Use clear ___ ___

• tell respondents ___ to search their memory

3. Provide useful ___ ___

• ___, ___, ___, calendars, or ___ can support recall when appropriate.

Good ___ cannot recover info that respondents cannot ___

Manageable, estimation, mental calculation, averaging, estimation, one, experiences, neutral, leading, emotionally, absolutes, always, never, make

Judgment: make the requested judgment ___

• reduce unnecessary ___ and make the judgment task manageable

1. Avoid unnecessary ___ ___

• do not make the respondents perform complicated ___ or ___

2. Ask ___ judgement at a time

• different ___ may require diferent answers

3. Use ___ framing

• avoid ___ or ___ loaded wording; be cautious with ___ such as “___” and “___”

A good item asks for a judgement respondents can realistically ___

Accurately, options, question, scale, clear, ordered, meaningful, non overlapping, editing, sensitive, neutral, privacy, options

Response: make it easy to report the judgement ___

• the response ___ should make the respondents judgement easy to report accurately

1. Match options to the ___

• the response ___ should represent the judgement being requested.

• How often? → never/rarely/sometimes/often/very often

• Not: strongly disagree/agree

2. Make categories ___:

• keep options ___, ___ and ___ ___

3. Reduce response ___

• for ___ topics, use ___ wording and protect ___ where possible

Even a good internal judgement can be lost if the response ___ do not fit it

Item, response process, diagnostic, represent, understand, access, judgment, options, wording, stage

Before piloting, review the whole ___

• use the ___ ____ as a final ___ check

1. Construct: does it ___ the intended construct or facet.

2. Comprehension: will respondents ___ it as intended ?

3. Retrieval: can they ___ the info needed to answer?

4. Judgment: can they reasonably form the requested ___ ?

5. Response: can they express that judgement using the available ___?

Clear ___ is not enough: every ___ has to work

Shorter, comprehension burden

Revision is not just ___ wording: it should reduce ___ ___!

Questionnaires, standardized, characteristics, experiences, behaviour, self report, personality, attitudes, symptoms, emotional, ratings, others, clinicians, parents, teachers, Hamilton depression rating, clinician, child behaviour, understood, remembered, judged, reported

___ are a common assessment method

• a ___ way to ask peopl about psychological ___, ___ and ___

1. ___ ___

• respondents answer questions about themselves

• Examples: ___, ___, ___ or ___ states

2. ___ of ___

• questionnaires can also be competed by ___, ___, ___ or other informants.

• Exmaples: the ___ ___ ___ scale: ___ rates a patient

• ___ ___ checklist: parent/caregver rates a child

The response process still matters: every questionnaire item must be ___, ___, ___ and ___ (the 4 stages)

Asked, how, structured, controlled, structured, format, scorable, combined, interpreted, standardized, efficiently, scoring, consistent, practical, inexpensive

Questions are response options work together

• a questionnaire standardizes both what people are ___ and ___ their answers are recorded

1. ___ questions: the same item or stimulus is presented in a ___ way

2. ___ responses: the response ___ defines how respondents can express their answer

3. ___ data: standardized responses can be ___ and ___ consistently

Why questionnaires are useful:

• ___ presentation of stimuli

• Many items can be administered ___

• ___ can be highly ___

• ___ for large groups and relatively ___

Format, construct, dichotomous, simple, efficient, degree, rating scales, degree, categories, frequency, intensity, agreement, variation, whether, how much, how often

Response ___ is a design choice

• different formats change what respondents can tell us about the ___

1. ___: check off relevant items

• Yes/No, True/false or checklist

• ___ and ___, but it captures little info about ___

2. ____ ___: measures the ___ of a response along an ordered set of ___

• used for ___, ___ or ___

• Captures more ___ than a simple yes/no response

• Design choice matters: number of response points, odd vs even number of points, inclusion of midpoint/neutral option, and category labels

• Example: during the last moth, how often did you have headaches? → not at all | occasionally | frequently | all the time

Dichotomous formats ask ___; rating scales ask ___ ___ or ___ ___

Relative, independently, forced choice, 2, compare, biases, desirable, distinct, desirability, restrictive, Q sort, categories, fixed distribution, priorities, self description, comparisons, ratings, relative standing, personality, California, relative, independent

Some formats ask respondents to make ___ choices rather than rathe each item ___

1. ___ ____

• choose between ___ or more alternatives

• ___ options relative to one another

• Can reduce some response ___

• Options should be similarly ___ and clearly ___

• Can reduce obvious ___ cues, but the response can feel ___

Example: which is more like you ?

A. I like to play to win Or B. I enjoy the company of others

2. ___ ___

• sort statements or characteristics into ordered ___

• Often uses a ___ ___

• Shows relative ___ or ___ ___

• Responses reflect ___ among items, not independent ___

• Useful when ___ ___ among characteristics matter

• Common for ___ assessment: the ___ thing test.

Example: group the symptoms according to how much they apply to you. Put one symptom in the least frequent category; one symptom in the most frequent category

Forced choice and Q-sort format require ___ choice rather than ___ ratings

Distorted, strategy, image, low effort, acquiescence, agree, reverse score, confuse, social desirability, faking, consequences, influence, Marlowe Crowne, response format, careless, random, sim, sacrifice, ignore, instructed response, bogus, improbable, time, fast, multiple indicators, quality

Even clear items can produce ___ answers

• some responses reflect ___, ___ management, or ___ ___ rather than the construct

1. ___: yea-saying or yes/agreement bias

• tendency to ___ regardless of content

• To combat it: use ___ ___ items

• Caution: reverse may ___ ppl

2. ___ ___/___: respondents may present themselves favourably or strategically

• especially true when the test has ___

• To combat it: measure ___ (example: use ___-___ measure of socially desirable responding)

• Or change ___ ___ (forced choice; q sort)

3. ___ and ___ responding

• respondents may ___, ___ or ___ items content

• To detect it:

• ___ ___ items: as respondents to select a specific option

• ___/___ items: include statements that attentive respondents should clearly reject (I was born before 1920)

• Response ___: unusually ___ completion can flag possible innatentive responding

• Best practice: use ___ ___ rather than treating any single check as definitive evidence

Item ___ includes conditions under which people can answer.

Study, cognitive interviews, ask, probe, diagnose, revise, pre testing, evidence

Do not wait until the full ___ to discover item problems

• ___ ___ reveal how respondents understand and answer draft items

1. ___ item: respondents answers draft question

2. ___: what did they think it mean ?

3. ___: where did the process break ?

4. ___: change wording or response options

Have to do this work before we get to the data.

___ turns item writing from guesswork into ___

Error, construct, response process, response format, define, response process, response format, pretest, revise

Good item writing is ___ prevention

• a useful score depends on the ___, the ___ ___, and the ___ ___ working together

1. ___: construct, facet, population and intended use

2. ___ ___: understand → retrieve → judge → respond

3. ___ ___: questionnaires, dichotomous, rating, forced choice, Q sort

4. ___ and ___: check bias and response quality; improve before full pilot