Psychology and Measurement: Validity, Reliability, and Item-Writing

A researcher intends to measure job performance. She defines it as the level at which an employee completes work tasks relative to a set standard. She models the construct so that three observed metrics combine to determine the performance score.

She believes the construct is formative, but her colleague believes it should be reflective. Which evaluation is most correct?

The researcher is correct. A construct defined as a level of execution relative to a standard is context-defined (formative).

A psychological measure intended to assess a relatively stable trait is administered to the same individuals on two occasions. Between administrations, several participants report being fatigued and distracted during the second session. The resulting scores differ considerably between time points for many individuals.

Which concept best explains the inconsistency in these observed scores?

Measurement Error

Kelsea, a third grade teacher, wants to measure how social each of her students is based on a personality scale she wrote. One question is: “On a scale of 1–10 how extroverted are you?”

Based on the information given, which maxim is Kelsea’s scale lacking?

Manner

Which of the following is NOT an example of classical conditioning? Based on ideal item-writing practices, which evaluation of this item is mist appropriate?

A. A dog salivates when it hears a bell previously paired with food
B. A student feels anxious when entering a room where they previously took a difficult exam
C. A person blinks when a puff f air is directed at their eye
D. A child learns to say please after being rewarded with praise

The item should be revised because it uses negative wording and includes a distractor that is not parallel in content.

John is on the item reduction stage of test development and has come across a scenario item that has underperformed during field testing, but seems to be salvageable. What would be the most important reason why he savages this item?

he will not meet the test blueprint requirements if he tosses out this item

A developmental psychology instructor creates a unit exam intended to assess students' understanding of childhood ADHD. The exam focuses almost entirely on diagnostic criteria and ignores other essential areas such as developmental course, common comorbidities, and evidence-based interventions. What does this indicate about the exam's validity?

The exam has poor content validity because it does not adequately represent the full range of important material related to childhood ADHD

Eric has just developed a four item survey with the intention of receiving qualitative information about how Louisiana Tech University undergraduate students encode important class information for exams. Which approach would provide the best feedback for Eric to establish content validity for his survey

The Think-Aloud Method

A research team is developing a three-dimensional personality scale measuring Conscientiousness(C), Neuroticism(N), and Openness(O). After running CVR study with 10 SMEs, they obtain the following results for one item

"I worry about making mistake". CVR(C): 0.4 ; CVR(N): 0.8; CVR(O):0.2

the minimum CVR threshold for 10 raters is 0.62. The item was written to measure Conscientiousness. Based on the CVR results, what is the correct decision about this item, and why?

Remove the item because its CVR for conscientiousness falls below the threshold, and its highest CVR belongs to a different construct.

A researcher develops a new anxiety scale that shows a strong correlation both with an established measure of anxiety and physical ability. WHat type of validity evidence does this pattern best represent

Convergent validity

A psychology student is developing a survey for a newly developed multi-dimensional construct, and skips the item reduction stage during the survey development process. What is the most likley consequence of this decision?

The number of underlying dimensions would be unknown

Use the scenario attached as an image to answer the question

Based on the scenario, which of the following best describes the main purpose of the process Dr. Alvarez used?

to ensure items represent the construct being measured

Use the scenario attached as an image to answer the question

Which issue best describes the problem identified by Dr. Alvarez?

The items includes content not a part of the construct being measured

Use the scenario attached as an image to answer the question

which of the following is likely true about the measure?

It has poor content validity because of issues with item interpretation and content consistency

Use the scenario attached as an image to answer the question

Which of the following is the BEST next step?

drop or revise items flagged as problematic by students

Use the scenario attached as an image to answer the question

According to the rules of survey item writing, what is the main problem with Haydens response

Hayden is reporting information even though he knows it is only a guess, not a true record

Use the scenario attached as an image to answer the question

Which conversational maxi is Haydens answer violating?

quality