Writing 1
Definition (#f7aeae)
Important (#edcae9)
Extra (#fffe9d)
Test | Scale |
Broader instruments developed using these scaling methods and item construction processes to measure something | Instruments used to measure traits, states, or abilities |
Measures ability | Also measure attitudes, beliefs and behaviours |
Usually have correct and incorrect answers | Used in a variety of settings, especially research |
Used predominantly in educational settings | - |
However, tests and scales share the same basic development process.
Test development process: Overview
Test development: Umbrella term for all that goes into the process of creating a test.
The creation of a good test is the product of applying established principles of test development.

Test conceptualisation: Principles Underlying Test Construction
Motives: What might lead a psychologist to think about designing or revising a psychological test?
No available tests for a psychological construct of interest (e.g., trait, state, attribute).
Existing tests may have poor psychometric properties.
Low reliability (inter-rater, internal consistency).
Low validity (construct, criterion-related).
Existing tests may have other undesirable qualities
Administration-related (time-consuming).
Interpretation-related (difficult to score).
When designing a new test, ask:
Questions about validity:
What is the test designed to measure?
What content will the test cover?
Questions about reliability:
What is the ideal format of the test?
Should more than one form of the test be developed?
Questions about the meaning of test scores:
What will test scores be used for?
Who will benefit?
Other:

Pilot Work: Preliminary research surrounding the creation of a prototype of the test.
May involve literature reviews, experimentation, and creation, revision, and deletion of preliminary items.
Test items may be "piloted" to evaluate their suitability.
Pilot work is necessary for published tests but not always for teacher-made tests.
Test construction:
Begins after the idea for a test has been conceived (step 1).
Involves:
Writing test items (re-writing/revising existing items)
Formatting items
Setting scoring rules
Otherwise designing and building a test.
Scales & Scaling:
Measurement (Scales):
The process of assigning numbers to individuals.
Particular attributes of individuals are reflected by some properties of numbers.
Scaling:
Scaling: The process of setting rules for assigning numbers in measurement.
The process by which a measuring device is designed and calibrated, and by which numbers (or other indices)— scale values—are assigned to different amounts of the trait, attribute, or characteristic being measured.
Measurement scales:
Attributes are more or less quantifiable:
Physical attributes: Easier to quantify, uses physical instruments.
Psychological attributes: More difficult to quantify, require indirect methods like questionnaires & observations.
4 types of measurement rules:
Nominal rule
Ordinal rule
Interval rule
ratio rule
How to combine scales & scaling to measure a psychological attribute
Sample behaviours:
Cheating on taxes if one has the chance.
Accepting a bribe in the course of one’s duties.
Assumptions:
People differ in the strength of their beliefs.
People can be ranked in terms of strength of belief (ordinal scale).
Measurement strategy:
Create an instrument that measures differences in belief strength.
Scaling methods:
Scaling is the process of assigning numbers to responses to calculate a test score.
There are many different methods of scaling; no "best" type exists.
Choice depends on variables measured, target group, and test developer's preferences.
Test-takers are presumed to have more or less of a given psychological attribute (trait, state) as a function of the test/scale score.
Summative scales:
Summative scales are a popular approach to scaling.
Scales on which a test-taker’s answers to test/scale items may be summed to produce an aggregate score.
Also widely known as “rating scales”.
This “aggregate score” is thought to indicate the strength of test-taker’s traits, attitudes or other characteristics.
These scales yield ordinal data.
Likert scales:
Provide test-taker with 5-7 possible responses along a agree to disagree continuum.
Easy to construct.
Easy for test-takers to understand - very straightforward.
Easy to be analysed Used extensively in psychology
True/False rating:
Binary choice scale (true/false, yes/no, etc.)
Faces scale:
How satisfied are you with this module?

Scaling methods that yield ordinal data:
Method of paired comparisons:
Test-taker has to choose one of two options (a statement, object, picture) on the basis of some rule.
The value of each option in each paired comparison is determined prior to test/scale administration by judges.
Comparative scaling:
Sorting or ranking a number of stimuli according to some rule (best to worst, most justifiable to least justifiable).
Categorical scaling:
Categorising a number of stimuli according to some rule (“best” & “worst”, “always justifiable” and “never justifiable”)
Writing Items:
The first step in writing items is to create an item bank or item pool.
Item pool: A collection from which final test items are drawn.
For multiple-choice tests, the initial pool might be twice the number of items needed for the final version.
Item ideas can come from the test developer's experience, experts, interviews with relevant individuals and literature reviews.
Advisable to create at least twice as many items as will be used in the final draft of the test/scale.
The process of test tryout, item analysis, and test revision will knock many of these items out as the test developer strives to maximise reliability and validity.
Item format:
2 types:
Selected-response format: Test takers have to select a response.
Constructed-response format: Test takers have to provide a response.
The choice of format depends on the test's purpose and the number of examinees.
Objective-subjective continuum:
Test items can exist anywhere along an objective-subjective continuum.
Selected-response format: Multiple choice, true-false items.
Constructed-response format: Short-answer, essay.
Constructed-Response Format:
Format 1: Completion items
Requires test takers to provide a word or phrase to complete a sentence.
Should be worded for a specific correct answer to avoid scoring problems.
Can also be called a short-answer item.
Poorly designed completion items can lead to scoring problems.
Format 2: Short answer items
Similar to completion but may be phrased as a question.
Answer is typically a word, term, sentence, or paragraph.
If more than 2 paragraphs, then item is likely to be an essay item
Drawbacks: Potential for inter-scorer reliability issues, as it is typically hand-scored.
Format 3: Essay item
Requires writing a composition in response to a question, demonstrating recall, understanding, analysis, and/or interpretation.
Useful for measuring depth of knowledge and ability to communicate ideas in writing.
Allows for creative integration and expression of what has been learned.
Drawbacks:
Essay items may not sample content as broadly as other formats.
Testtakers may "bluff".
Scoring is time-consuming and subjective.
Potential for inter-scorer reliability issues.
May rely too heavily on writing skills.
Selected-Response Format:
Matching Items:
Useful for evaluating recall of related facts, especially when many facts on a single topic are involved. Involves matching items from two lists (premises and responses).
To prevent guessing, the response list should have more items than the premise list, or directions should state responses can be used multiple times or not at all.
Should be homogeneous and relatively short.
Test-takers only need to recognise, not recall or devise.
Test-taker presented with premises (left column) and responses (right column).
Task is to determine which premise is associated with each response

True/False item:
Also known as binary choice format.
Varieties include: yes/no, agree/disagree.
Must contain a single, unambiguous idea & be short.
Pros:
Easier to create (construct) and score than multiple-choice items.
Can sample a lot of content quickly.
Cons:
High susceptibility to guessing, especially for "test-wise" students. Response can be based on guess.
Wording can lead to different interpretations. Only usable when unqualified dichotomous responses are possible.
Multiple choice:
3 elements:
A stem
A correct option
Multiple incorrect alternative options
There must be one correct option and all options must be plausible and grammatically comparable.
Pros:
Can be administered quickly and has broader coverage.
Cons:
Limits the responses which can be made.
Susceptible to guessing if items are poorly written or patterns are discernible.
More Complex Multiple Choice Formats
Classification: The test-taker classifies a person, object or condition into one of several categories designated in the stem.
If-then conditions: The test-taker must decide the consequence of one or more conditions being present.
Multiple coditions: The test-taker uses the two or more conditions in the statements listed to draw a conclusion.
Multiple true/false: The test-taker decides whether one, all, or none of the two or more conditions or statements listed in the stem are correct.
Oddity: The test-taker indicates which option does not belong with the others.
Objective-subjective continuum:
Test items can exist anywhere along an objective-subjective continuum.
Objective end:
Consist of multiple choice items, true-false items.
Easily scored, has one correct answer, easily analysed.
Only provides information about correct answer and can be a result of guessing.
Subjective end:
Consist of essay / open ended questions.
Responses can be unique, in-depth, or personal.
Difficult to quantify and analyse.
Writing strong test items:
Items must be clear and unambiguous.
Should not be double barrelled.
Each item only ask about one aspect.
Should not use subjective perception of time.
Be specific about the time (week, month).

Test Tryout:
Involves administering a preliminary form of the test.
This administration is conducted on a representative sample of test-takers.
Goal: To administer the test under conditions that simulate the conditions under which the final version of the test will be administered.
Purpose:
The primary purpose is to collect data from the tryout administration.
These data are used to analyse test-takers’ performance on the test as a whole and on each individual item.
This analysis, item analysis, helps in making judgments about which items are effective, which need revision, and which should be discarded.
Guidelines for test tryout:
Sample selection:
Conduct the tryout on participants similar to the intended test-takers.
Sample size:
Aim for 5–10 participants per item on the test to reduce chance effects in analysis.
More subjects = stronger data reliability.
Testing conditions:
Ensure tryout conditions mirror standardized test conditions as closely as possible.
This includes all instructions, time limits, and the atmosphere at the test site.
Ensuring consistent conditions helps guarantee that differences in response are due to the items themselves, rather than external factors.
This ensures that the validity and reliability of the test items are retained.
Outcome of Tryout and Subsequent Stages:
Data collected during the tryout (scores & responses) are subjected to item analysis.
Item analysis can be quantitative (using statistical procedures) or qualitative (using nonstatistical procedures).
Based on the item analysis, test revision occurs, where the test's content or format is modified to improve its effectiveness. Items might be revised, deleted, or new ones created.
Test Tryout as an Iterative Process:
The process of revision, tryout, and item analysis may be repeated until the test is satisfactory.
A revised version of the test is typically tried out with a new sample of testtakers.
Data from this administration are analysed, and further revision occurs if necessary.
Once the test is deemed satisfactory after tryout and analysis, the data from the final administration (often the second sample) can be used to develop norms, and the test is considered standardized.
Pilot work:
Pilot work or a pilot study refers to preliminary research surrounding the creation of a prototype test.
This can involve experimenting with items, conducting interviews, and reviewing literature to determine the best way to measure the target construct.
While pilot work precedes formal test construction, the need for further pilot research is always possible based on feedback and results.
Pilot work is considered a necessity for tests intended for publication and wide distribution.