Overview: Emphasizes choosing answer “C” yielding correct answers approximately 25% of the time.
Recording Notice: All lectures are audio recorded.
Source Reference: All content sourced from Salkind, Tests and Measurement 3e. SAGE Publishing (2018).
Characteristics of Multiple-Choice Items
Preferred Testing Method:
Widely utilized for testing achievement-oriented outcomes.
Common in fields such as:
Chemistry
Biology
Psychology
Applied as the written component in tests like the Red Cross CPR test and various nursing board exams.
Often referred to as “objective items”.
Benefits of Multiple-Choice Items
Scoring and Analyzing:
Easy to score: Automated scoring possible through scanning technologies.
Easy to analyze: Rich data sets available for tester feedback.
Flexibility:
Adaptable in terms of learning outcomes it assesses.
Bloom’s Taxonomy:
Can be designed to fit any level within Bloom's taxonomy, which categorizes educational goals.
Structure of Multiple-Choice Questions
Anatomy of MC Questions:
Comprised of three essential parts:
Stem: The primary question or statement posed.
Correct Alternative: The correct answer choice.
Distracters: The incorrect answer choices, or “decoys”.
Example Question:
The example illustrates a question regarding logical fallacies:
“Presenting a very weak form of an opposing argument is one component of…”
a. appeals to pity.
b. testimonials.
c. creating false cause.
d. a straw person.
(This example is modified from Halpern, 2013.)
Guidelines for Writing Multiple-Choice Items
Format: Write alternatives separately, one after the other.
Learning Outcomes Alignment: Each question must align clearly with intended learning outcomes.
Avoid Negatives: Recommended to steer clear of negatives in the stem or answer alternatives.
Not Recommended: “Which of the following statements is not true?”
Recommended: “Which of the following statements is false?”
Positioning: Vary the position of the correct answer to reduce impact of guessing.
Clarity and Consistency in MC Items
Stem Writing: Ensure stems are self-contained, clear, and precise.
Example of Poorly Written Question:
“New York City is the site of the next Olympics and”
A. has a new stadium for track and field.
B. is building a new stadium for track and field.
C. will be using only the Jets stadium in New Jersey.
D. hasn’t yet completed plans for where the stadium event will take place.
Grammar and Consistency: Use correct grammar and maintain consistency throughout.
Pitfalls in Writing Multiple-Choice Items
Negatives, Absolutes, and Qualifiers:
Among potential pitfalls, these should be avoided.
Example: “Not only do cicadas come every 17 years, but they never arrive”
a. during a rainy season.
b. only if the temperature is sufficiently warm.
c. whenever the ground is soft enough for them to emerge.
d. after June 1.
Distractors Quality: Ensure distractors are plausible and independent of one another; they should not give clues to the correct answer in other items.
Advantages of Multiple-Choice Items
Positives:
Useful for assessing learning at nearly any cognitive level.
Clear and straightforward nature minimizes writing demands on test takers.
Guessing effects are reduced in comparison particularly to true-false items.
Easy to score with reliable results, enabling effective test analysis.
Disadvantages of Multiple-Choice Items
Negatives:
Guessing can lead to skewed scores.
Extensive time required to formulate quality questions.
Developing good multiple-choice items is challenging.
Limited expression of creativity or unique responses from test takers.
Top-performing students may demonstrate more knowledge than what is assessed by the items.
Question Review and Modifications
Evaluating the need for question modification involves considering question difficulty and clarity.
Example Question: - “The number of questions we can expect students to complete in one hour is”
A. 3
B. 50
C. 125
(Modified from Salkind, Tests and Measurement 3e.)
Different Types of Multiple-Choice Items
Best-Answer Items: May contain multiple correct answers but only one is considered the best choice.
Rearrangement Items: Participants are tasked with arranging a set of items in sequential order.
Interpretive Items: Involve presenting a passage followed by questions based on that passage.
Substitution Items: Alternates are provided, and test takers select responses they believe correctly answer the question.
Analyzing Multiple-Choice Items
Item Analysis: The process of assessing multiple-choice items through a thorough examination of their effectiveness.
Components of Item Analysis:
Difficulty Index: Measures how many test-takers got an item correct.
Discrimination Index: Indicates how effectively items discriminate between those who know the material and those who do not.
Calculation of Item Analysis Metrics
High and Low Groups:
Create high and low scoring groups from test-taker responses (top 27% and bottom 27%).
Examination is conducted per item, not on full test.
Difficulty Index Formula
Difficulty Index (D):
$D$ = $ rac{h}{T}$
Where $h$ = number of correct responses in the high group.
$T$ = total responses to the item.
Lower percentages indicate higher difficulty and vice versa.
Example Calculation of Difficulty Index
Sample Data:
Total Responses = 50
Correct Responses (High Group) = 22
Correct Responses (Low Group) = 14
Calculating D:
$D = rac{22 + 14}{50} = 0.72$ or 72% (indicating how many got it correct)
Discrimination Index
Definition: Measures how effectively an item distinguishes between high and low-scoring groups.
Formula:
$d = rac{h - l}{T/2}$
Where $h$ = correct responses in high group, $l$ = correct responses in low group, $T$ = total number of correct responses.
Index Range: Values range from 0.00 (no discrimination) to 1.00 (perfect discrimination). Negative numbers reflect a poorly constructed item.
Characteristics of a Perfect Item
Ideal Conditions:
Difficulty level $D$ = 50%: Half the test-takers responded correctly.
Discrimination level $d$ = 1.00: High group scores correctly while the low group scores incorrectly.
Plotting Difficulty and Discrimination Indexes
A graphical representation outlines the relationship between difficulty level and discrimination effectiveness where optimal items appear in the area where D equals 0.50 and d equals 1.00.
Interpreting the Difficulty and Discrimination Results
Item 1:
D = 25% (difficult), d = 0.50 (positive discrimination).
Item 2:
D = 75% (easy), d = 0.50 (positive discrimination).
Item 3:
D = 75%, d = -0.50 (negative discrimination): indicates potential issues with distractors.
Examination of Items: Critical analysis of items to establish their efficacy and the need for any revisions.
Analysis of Item Alternatives
Quality of Alternatives: Assessment of choices made available in each multiple-choice item is crucial. Effective distractors should differentiate knowledgeable students from those who lack familiarity. No formulas or complex statistics are required for this analysis.
Response Patterns: Each item displays response patterns for upper and lower scorers, allowing educators to gauge the effectiveness of each alternative.
Example Analysis of Item Responses:
Analysis revealing how distractor patterns correlate with test performance for items 4 and 5.