PSYC3130 Lecture8_Slides

3 reasons we develop new tests

•Meet the needs of a special group of test takers (e.g., special population with a disability, pediatric population, geriatric population)

•Sample behaviours from a newly defined test domain (e.g., new definition of cyberbullying)

•Improve the accuracy of test scores for their intended purpose (e.g., test had low reliability or validity)

Step 1:

Defining the Testing Universe, Audience, & Purpose

Defining the Testing Universe

• Prepare a working definition of the construct (operational definitions)

• Locate studies that explain the construct

• Locate current measures of the construct

Defining the Target Audience

Tests should be designed with specific users in mind

•Make a list of characteristics of persons who will take the test, particularly those characteristics that will affect how test takers will respond to the test questions

–E.g., age range, reading level, disabilities, language, honesty

Defining the Test Purpose

Defining the Test Purpose includes not only what the test will measure, but also how scores will be used.

–Will scores be used to compare test takers (normative approach), will they be used to compare a test taker standing on one trait with their own standing on other traits (ipsative approach), or to indicate achievement (criterion approach)?

–Will scores be used to test a theory or to provide information about an individual?

•E.g., Are people with depression likely to suffer from anxiety vs. How smart is that individual?

•If used to test theory, the purpose is to deliver a distribution of scores (shorter test with lower reliability may be sufficient); If used to provide info about individual, accuracy is most important.

Step 2:

Developing a Test Plan

Test Plan

specifies the characteristics of the test, including an operational definition of the construct and the content to be measured, the question format, and the administration and scoring of the test.

•Many researchers think they have a clear idea of what they want to measure only to find out that their ideas or too vague or specific.

Defining the Construct and the Content to Be Measured

Define and operationalize construct in terms of observable and measurable behaviours.

•Provide boundaries for the test domain (what should and shouldn’t be included)

•Specify approximate number of questions needed to sample the test domain.

Objective test format

has one response that is designated as “correct” or that provides evidence of a specific construct (e.g., multiple-choice, true/false, fill-in-the-blank)

–Preferable

Subjective test format

do not have a single response that is designated as “correct” (e.g., projective tests, open-ended or essay questions, interviews)

Administering and Scoring

Test developers must specify how to do so

•Will test be administered in writing? Orally? By computer?

•How much time is needed to complete the test?

•Will it be administered to groups or individuals?

•How will it be scored and what types of data is it expected to produce?

–Determine a model of scoring

Cumulative model of scoring

most common method for determining an individual’s final test score.

•Assumes that the more a test taker responds in a particular fashion the more they exhibit the attribute being measured (e.g., more “correct” answers, or endorses higher numbers on a Likert scale)

•Correct responses or responses on Likert scale are summed.

•Yields interval data that can be interpreted with reference to norms.

•E.g., IQ test, Stress scale

Categorical model of scoring

used to place test takers in a particular group or class

•E.g., a particular pattern of responses may suggest diagnosis of a certain psychological disorder

•Typically yields nominal data because it places test takers in categories (e.g., the # of true vs. false, agree vs. disagree)

Ipsative model of scoring

the test taker is usually presented with two to four statements in a forced-choice format

•Most often used in personality assessment

•E.g., test taker must decide which of the items is “most like me” and which is “least like me” (choices must be equal in desirability)

•Only provides information regarding where test takers stand relative to themselves on the measured constructs (cannot make comparisons between individuals)

Step 3

Composing the Test Items

Multiple choice

this format is familiar, many people use it for pre-employment tests, standardized achievement tests, and classroom tests.

–consists of a stem (question or partial sentence) and a number of responses (4-5) of which only one is correct

–the wrong answers are called distractors because they may appear correct & must be realistic

–important to clearly differentiate the correct response from distractors

Multiple Choice pros and cons

 Pros

•More answer options (4-5) reduce the chance of guessing that an item is correct

•Easily scored (can be scored by computer) 

Cons

•Test takers may obtain credit by guessing (can be offset by presenting a large number of items– but this increases reading time)

•Measures narrow facets of performance

•Transparent clues (e.g., verb tenses or letter uses “a” or “an”) may encourage guessing

•Difficult to write four or five plausible choices

•Takes more time to write questions– limit use of “none of the above” or “all of the above”

Objective items - true/false

asks “Is this statement true or false?”

–Typically used in educational testing and some personality testing

–Test taker can again gain some advantage by guessing (especially on short tests that contain a small number of items)

–Equal # of true and false answers should be written

–Avoid including more than one idea in the statement

Objective items - forced choice

used for personality and attitude tests rather than for knowledge tests.

–Requires test taker to choose one of two or more words or phrases that appear unrelated but are equally acceptable.

–Although words or phrases appear unrelated, the test developer must have empirical evidence that the responses yield significantly different responses from different types of people.

–Pro: Less bias, more difficult to “fake” a test or guess answer.

–Con: What if there is no valid choice? Lower face validity.

Subjective items

are less easily scored and interpreted but give test taker fewer cues and wider areas for response– often used in educational & organizational settings.

Essay Questions

usually general in scope and require lengthy written responses by test takers.

–Similar to open-ended surveys

–Scored on the basis of how closely response matches a predetermined correct response

–Pro: Provide freedom of response that facilitates higher cognitive behaviours; Allows test takers to demonstrate learning

–Con: Writing skills may influence scorer judgements; Subjectivity 

Interview Questions

general in scope and the interviewer decides what is a “good” or “poor” answer.

–Traditionally used for employment

–Questions are general in scope

–Interviewer decides what is a “good” vs. “poor” response

–Pro: Allows for freedom of response, wide range of responses

–Con: May induce anxiety; Subjectivity in scoring

Projective techniques

a subjective format used often for diagnosing mental health disorders (personality).

–Uses highly ambiguous stimulus to elicit an unstructured response

–Test taker “projects” their perspective and perceptions onto a stimulus

–May contain pictures, written/spoken words

–Responses may be verbal or nonverbal (drawings)

–E.g., Rorschach Inkblots Test, Thematic Apperception Test (TAT)

Sentence completion

A projective technique that presents an incomplete sentence that test taker must complete (sometimes timed- must complete as quickly as possible)

–E.g, Rotter Incomplete Sentence Blank (RISB)

–3 forms: school, college, adult

–Includes 40 incomplete sentences (timed)

–Assumes that responses will reveal internal desires, fears, struggles

–Scored by comparing responses with those supplied by test developer to award points or identify a trait

Pros and cons of projective tests

Pros:

•Allow for wide range of responses, greater depth

•Can help reveal unconscious cognitions, feelings, beliefs

•Can aid in therapy

Cons:

•Subjective interpretation (high rates of judgment translates into high error– evidence of interrater reliability and clear scoring guidelines are key)

•Time consuming

•Poor cross-cultural validity

Performance assessments

require test takers to directly demonstrate their skills and abilities to perform a group of complex behaviors and tasks.

–Setting in which these tasks are demonstrated is made as similar as possible to the conditions that will be found when the tasks are actually performed.

–E.g., audition of musician trying out for a band

–In employment setting, performance assessments are often called “work samples”

Portfolio

a collection of work products that a person gathers over time to demonstrate his or her skills and abilities in a particular area.

–E.g., artistic portfolio, teaching dossier

–Scoring process should be standardized so that results are comparable across different people

Important point: complex item formats are all are subject to and evaluated by the same technical standards as any other test. All the issues regarding reliability/precision and validity evidence also apply to these types of measures.

Response bias

Tests are subject to error and one form comes from the test takers themselves.

 

Response bias results in response sets/styles such as:

•social desirability

•acquiescence

•random responding

•faking

Response Sets/Styles

Patterns (or styles) of responding that result in false or misleading information and limit the accuracy and usefulness of test scores

Reasons for misleading information

•Information requested is too personal

•Answer items carelessly

•May feel coerced into completing the test or not motivated

•Believe this is how they are supposed to answer

Social Desirability

Some test takers choose socially acceptable answers or present themselves in a favorable light.

•People may not attend to the accurate answer (trait being measured) but attend to the social acceptability of the statement.

•This represents unwanted variance – it might be a problem of the item itself

–E.g., when writing multiple-choice questions, it is important to balance the social desirability of the correct response and the distractors

•Solution: examine whether responses on your test correlated strongly with a social desirability scale

Acquiescence

Tendency to agree with any ideas or behaviours presented

•E.g., labelling each statement on a true/false test as true

•More prevalent in societies that value deference and politeness

•Can be seen on tests assessing attitudes, personality, public opinion polls

•Solution: balance items for which the correct response would be positive vs. negative (reverse scoring)

Random Responding

Responding to items in a random fashion by marking answers without reading or considering them.

•Test takers are unwilling or unable to respond to test items accurately.

•May occur when test takers lack the necessary skills (e.g., reading/writing) to take the test, don’t want to be evaluated, lack motivation or attention span necessary to complete task.

Faking/Malingering

Some test takers may try to answer items in a way that will cause a desired outcome or diagnosis.

•May “fake good” (e.g., in employment settings) to create a favorable impression

•May “fake bad” (e.g., in clinical or forensic settings) to appear worse than they really are

Reasons:

•Cry for help

•Want to plea insanity in court

Potential solutions for faking/malingering

–Use of subtle questions that are difficult to fake because they don’t have high face validity.

–Include special items that are scored separately from the test (”faking scales”) to flag those who score high (e.g., “I get angry sometimes”).

–Use two different responses to two similar questions.

–Use of forced-choice format.

–Use multiple assessment methods (other than self-report).

–Repeat testing.

Bosus Statement (BS) scale

•Asks test takers how familiar they were with nonexistent concepts or whether they had ever performed “nonexistent” tasks (e.g., “I am familiar with the Automatic Textbook Citation Creation System”)

•A high score on a faking scale identifies test takers who are uncooperative, responding randomly, or faking.

Bender Gestalt test

•Visual-motor functioning & perception skills

•Used in assessment of brain injury

Rey 15-Item Memory Test (RMT)

Dot Counting test

Time to count grouped vs. ungrouped dots

Rule of thumb for writing effective test items

Write twice as many items as you expect to use in the final test.

Writing Effective Items - Multiple-Choice and True/False

•Avoid using negative stems and responses: Instead of asking, "Which of the following is NOT true?" it is better to ask, "Which of the following is FALSE?"

•Make all responses similar in detail and length.

•Make sure the item has only one answer that is definitely correct or "best.”

•Avoid determining words such as always and never (better to use sometimes and often)

Writing Effective Items - Essay and Interview Questions

•Use essay items appropriately: most effective for assessing higher order skills such as analysis, synthesis, and evaluation (not for simple recall)

•Consider the time necessary for response– let the test taker know the expected length of response.

•Prepare an answer key– list other possible responses along with scores based on the appropriateness of the responses.

•Score essays anonymously (decreases bias).

•Use multiple independent scorers

Sampling

•Objective formats provide more opportunities to sample testing universe as a result of less time & effort for completion.

•Covering wide range of topics can increase content validity of test.

Construction

•Objective tests require extensive thought and development time.

•Subjective tests require fewer items (easier to construct)- more appropriate for assessing higher-order skills.

Scoring

•Scoring objective items is simple and can be done by computer (high reliability and accuracy)

•Scoring subject items is time consuming and difficult (subjectivity- need two independent scorers to avoid bias)

Response sets

•Objective tests allow guessing, higher rates of socially desirable responses or acquiesce to positive statements

•Subjective tests are more difficult to “fake” but may be influenced by poor verbal or writing skills

Step 4

Writing Administration Instructions

Administrator Instructions

specify the testing environment to decrease variation or error in test scores

•Group or individual administration

•Requirements for location (e.g., quiet, privacy)

•Required equipment (computer, pencil)

•Time limits or approximate completion time

•Script for administrator and answers to questions test takers may ask

•Credentials or training required for test administrator

Instructions for Test Taker

Test administrator delivers instructions for the test taker orally by reading a prepared script written by the test developer.

•Instructions also appear in writing at the beginning of the test or test booklet

•Each item should be preceded by specific instructions

•Should be simple, precise and written at low reading level

Scoring Instructions

Scoring instructions and test key ensure that each person who scores the test will follow the same process.

•Must also explain how the test scores relate to the construct the test measures (e.g., what does a high score mean vs. what does a low score mean?)

Step 5

Conducting Pilot Tests

Pilot Test

a scientific evaluation of the test’s performance to investigate reliability and validity

–Cannot assume the test will perform as expected

–Goal is to administer test to sample from the target audience, analyze data and revise test to fix any potential problems

–Sample should be large enough to conduct statistical tests and setting of pilot test should mirror planned test setting

–Test takers may be interviewed about their experience

–May be necessary to conduct multiple pilot studies