Questionnaire Design Pitfalls & Best Practices

Course & Session Context

  • Graduate-level course on research methodology; current unit: constructing questionnaires/surveys.
  • Purpose of the session:
    • Debrief on an earlier reflection exercise about coping with tragedy and being present for clients.
    • Clarify an administrative issue: Module 2 video lost its multi-item quiz; students should upload a screenshot of the single remaining question for credit.
    • Launch Module 4 assignment details: final questionnaire due in Module 4 along with pilot-test data.
  • Personal note from instructor: wearing new clear aligners as part of a TMJ treatment journey—shared to “arrive with a clean, clear smile.”

Assignment Logistics & Timeline

  • Deliverables
    • Draft questionnaire now (Module 3).
    • Pilot test between now and Module 4 submission date.
    • Submit final questionnaire + pilot feedback + preliminary data in Module 4.
  • Pilot requirements
    • Must administer to at least one set of laypersons (friends/family) to check readability and comprehension.
    • Optional but recommended: swap pilots with another student group; avoid over-burdening any single cohort member—decline politely if you receive multiple requests.
  • Software options
    • Google Forms: simplest; meets most needs.
    • Qualtrics: free with Baylor login; better for complex logic (branching, piping, forced display logic). Recommended if survey contains conditional branches or comparison groups.

Big Picture: Why Pilot?

Pilot testing uncovers

  • Misinterpretations of wording (literal vs intended meaning).
  • Missing response options and overlapping categories.
  • Technical glitches (required fields, branching errors, mobile display issues).
  • Data-cleanliness concerns (e.g., unexpected text in numeric fields).

Common Questionnaire Hazards (with Illustrative Examples)

1. Basic Grammar & Mechanics

  • Verify punctuation: statements end with a period, questions with a question mark; partial stems (e.g., “Because …”) have neither.
  • Spell-check manually: survey tools’ spell-checking is often disabled in question text boxes.
  • Case study: “Asynchronous material prepares me for synchronous sessions” lacked a period and misspelled “synchronous/asynchronous.”

2. Missing Instructions

  • Always tell participants what to do (rate, select, type, rank, etc.).
  • Add an introductory phrase if questions appear in a block sharing directions.
    • Example fix: “Rate your agreement with the following statement.”

3. Jargon, Ambiguity & Literacy Level

  • Define or remove technical terms (e.g., “auditory study environment”).
  • Replace vague phrases like “most conducive to learning” with precise constructs (better focus? higher grades? greater retention?).

4. Wrong Question Type / Double-Barreled Items

  • Each item should measure one construct.
  • If you need explanation and numeric rating, split into (a) forced-choice and (b) open-ended follow-up.
    • Example: “Explain why you selected that option” was glued to a Likert question—creates two questions in one.
  • Use branching or display logic when follow-ups depend on prior answers.

5. Interesting but Irrelevant Questions

  • Every question must tie directly to the study’s research aims, demographic descriptors, or necessary screening.
  • Remove “nice to know” items that bloat length and lower completion rates.

6. Producing Murky / Unusable Data

  • Preference questions without context (e.g., “Do you prefer synchronous or asynchronous?”) yield meaningless data if underlying reasons aren’t captured.
  • Open-ended text boxes alone create coding burden; pair with structured choices + an "Other (please specify)" field.

7. Overlapping or Incomplete Response Options

  • Ranges must be mutually exclusive and exhaustive.
    • Bad: 1{-}2\text{ days},\;2{-}4\text{ days} (overlap at 2).
    • Good: 1\text{ day or less},\;2{-}3,\;4{-}5,\;6{-}7.
  • Example: “Three to five days” vs “One to two days” vs “Day it’s due” vs “Hour it’s due” omitted 6–7 days and had overlap at “day/hour.”

8. Uneven Scales (Response Cliffs)

  • Scales must be symmetric around a neutral point with equal intervals.
  • Faulty example: Neutral | Slightly Effective | Effective | Very Effective → only one negative option.
  • Provide balanced anchors: \text{Very Ineffective},\;\text{Ineffective},\;\text{Neutral},\;\text{Effective},\;\text{Very Effective}.

9. Improper Use of “Other” Field

  • In forced-choice items, “Other” should be available only when legitimate options are unforeseeable, else force a valid choice.
  • Do not add “Other” on numeric or scalar items; off-scale answers cannot be analyzed quantitatively.

10. Assumptive Wording

  • Avoid presupposing behaviors or contexts.
    • “Which tool helps you manage time?” assumes respondent already uses a tool.
    • Better: “Which of the following tools, if any, do you use to help manage your time?”

11. Excessively Open-Ended Prompts

  • “List three words describing group projects” yields heterogeneous text (single words, phrases, hyphenations).
  • Hard to code; consider structured lists plus an “Other” textbox.

12. Data Entry & Coding Burden

  • Simulate data-entry: copy a sample of responses into rows/columns to see if analysis will be straightforward.
  • If the answer cannot be coded into tidy columns, refine the question.

Illustrative Fixes & Redesigns

  1. Original: “Choose the auditory study environment most conducive to your learning.”
    Fix: “Choose the study environment you use most often. Options: Loud (TV/music playing), Moderate background noise (coffee shop), Quiet (library), Silent (noise-canceling headphones).”
  2. Original: “During your higher education pursuits (undergraduate and/or graduate school) please explain your opinions on group work as favorable …” Fix split into:
    • Q_1 (Likert): “During graduate studies, how would you rate your overall attitude toward group work?”
    • Q_2 (Open-ended): “Please describe one factor that most influenced the rating above.”
  3. Original: “What methods do you use to manage your time? paper/digital planner, paper/digital calendar…”
    Fix: Break each medium into discrete options; allow multiple selection and an ‘Other (app name)’ textbox.
  4. Scaled example: “On a scale of 1 (not beneficial) to 10 (very beneficial), rate how group projects enhance your learning.”

Good Question Characteristics (Positive Models)

  • Clear construct & timeframe: “How many days per week do you interact with course materials?” with answer set {0,1,2,3,4,5,6,7}.
  • Balanced numeric scale: 1{-}10 slider with labeled anchors.
  • Exhaustive list without overlap.
  • Directions embedded (“Select one,” “Check all that apply,” “Rate each statement”).
  • Avoid brand names unless brand is central to research; e.g., replace “Google Calendar” with “Digital calendar (e.g., Google, Outlook).”

Qualitative vs Quantitative Items

  • Qualitative (open text) useful for depth, motivation, unforeseen themes.
  • Quantitative (forced choice, Likert, numeric) needed for statistical comparisons.
  • Use mixed-methods design wisely: open-ended follow-ups sparingly; plan coding scheme in advance (Module 3 coding workshop upcoming).

Practical / Ethical Considerations

  • Reading level: target \leq 8^{\text{th}}-grade for lay audiences unless surveying professionals.
  • Cultural inclusivity: avoid idioms (“hit the books”), region-specific slang.
  • Privacy: skip collecting identifying info unless essential; if collected, store separately.
  • Participant burden: keep survey length short; remove extraneous items; pilot to gauge completion time.

Instructor Tips & Reminders

  • Force answers when the item is essential; otherwise allow skip.
  • For branching logic, Qualtrics > Google Forms.
  • When unaided recall is impossible (e.g., exact dates), offer ranges.
  • Always include an “I don’t know / Not applicable” when some respondents truly lack the information.

Connections to Previous & Future Modules

  • Module 1: Research questions & hypotheses set the blueprint; only include survey items that trace back to these.
  • Module 3 (next week): Qualitative coding techniques—will demonstrate how open-ended answers are analyzed (content analysis, coding tree, inter-rater reliability). No additional class time for questionnaire edits next week.
  • Module 4: Final questionnaire, pilot report, and preliminary descriptive stats due.

Real-World Relevance

  • Poorly designed surveys waste resources and can misinform policy/clinical practice.
  • Balanced, well-piloted questionnaires enhance evidence-based decision-making in occupational therapy and other disciplines.

Action Checklist Before Module 4 Submission

  • [ ] Review every item for grammar, spelling, clarity.
  • [ ] Verify response options are exhaustive & non-overlapping.
  • [ ] Ensure scales are even and anchored.
  • [ ] Remove brand names unless integral.
  • [ ] Explicitly state instructions for each item or block.
  • [ ] Mark required items appropriately; include NA/Other where justified.
  • [ ] Conduct at least one layperson pilot; collect feedback on confusion points.
  • [ ] Simulate data coding; confirm each response fits intended analysis.
  • [ ] Update survey software settings (branching, validation, display logic).
  • [ ] Save/print questionnaire and pilot summary for Module 4 submission.