Lecture 5: sources of measurement error and how to reduce them

true or false: validity and reliability are entirely related

false: they are partially related and partially independent

true or false: observed scores are a reflection of the variable/construct being measured

false: there is always some degree of error present

observed score = true score + error

what are the types of measurement error? (2)

• random: aspects beyond experimenter’s control

• systematic: experimenter can try to reduce the error

explain the “random” error

• aspects beyond the experimenter’s control

• it changes randomly between each trial, unpredictable

• the experimenter can try to distribute the error across conditions

• ex: the amount of sleep participants had

explain the “systematic” error

• experimenter can work on reducing the error

• error doesn’t change randomly

• ex: the way we give instructions

(topic of this set of flashcards)

what are the common sources of systematic error? (4’

• participants

• equipment/instructions

• testing environment

• experimenter bias and scoring guidelines

define “reactivity”

when the participant modify their natural behaviour because they know that they are being measured

define “response set”

being ready to answer a certain way

what could affect the reactivity and response set? (6)

• mood

• motivation

• fatigue

• memory

• practice

• knowledge

define “good participant role”

participant who wants to produce answer that supports the hypothesis

true or false: we want “good participants role” as participants in our study

false: they tend to answer what supports the hypothesis… but we don’t want that. we want faithful participant role

define “negativistic participant role”

participant that will answer contrary to the hypothesis

define “apprehensive participant role”

participant that belies that they will be judged and seek social desirability

define “faithful participant role”

participant that will follow the instructions, without biases or predisposition to positive or negative

what are the subject roles and which one do we want in our study? (4)

• good: wants to produce answers that support the hypothesis

• negative: says the opposite of the hypothesis

• apprehensive: know that they will be judged, wants social desirability

• faithful: will follow instruction, participant we want

define “subject roles” or “participant roles”

mindset that the participant brings to the task (there are 4 of them)

what are the acquiescence response biases? (2)

• acquiescence: tendency to say “yes”

• disagreeing: tendency to say “no”

true or false: high or low scores are always a sign of response bias

false: not always, but it can be hard to distinguish between real and bias

how can you try to avoid the acquiescence response bias? (2)

• phrase things that are seen as undesirable on a more acceptable way

• include items that measure social desirability

how can equipment change the results? (3)

• sensitivity: does it detect subtle changes

• instructions: are they understandable

• intrusiveness: is the experimenter present? that can affect the results

what are the character is of good instructions? (4)

• clarity

• appropriateness

• length

• simple vocabulary

what are the range effects? (2)

• ceiling effect: equipment doesn’t detect change at the high end (values could be higher)

• floor effect: equipment doesn’t do text change at the low end (values could be lower

define “ceiling effect”

equipment isn’t sensitive enough and doesn’t detect a difference at the high end of the scale

define “floor effect”

equipment isn’t sensitive enough and doesn’t detect a difference at the low end of the scale

what could affect the testing environment? (3)

• comfort: room too hot?

• presence of others/social facilitation

• distractions: noise, interruptions

define “experimenter bias”

measurement is influenced by the experimenter’s expectations or beliefs regarding the outcome

true or false: all biases are intentional

false: they can also be intentional

what are the ways an experimenter can influence a participant? (5)

• paralinguistic cues: variation in tone of voice

• kinesthetic cues: body posture, facial expression

• verbal: reinforcement of expected or desired responses

• misjudging participant’s response in the direction of the expected results

• not recoding responses accurately

how can you reduce experimenter bias?

by standardizing or automating the experiment

how can you automate an experiment? (2)

• single-blind: participant doesn’t know in which condition they are, but experimenter does

• double-blind: both the participant and the experiment don’t know in which condition the participant is in (a research assistant will do the assignment)

define “single-blind”

participant doesn’t know in which condition they are, but experimenter does

define “double-blind”

the experimenter and the participant don’t know in which condition the participant is in

explain the experiment that proved the experimenter bias

• two groups of students. one group had maze-bright rats (smart) and the other had maze-dull rat (not that smart)

• all rats were the same

• stem dents had to teach the rat to go to the grey side of the box through reinforcement

• students with bright rats reported that their rat learned quickly compared to dull rats

→ students unconsciously influenced the performance of their rat

what are the things you can standardize to minimize the error? (4)

• participants

• test protocol

• testing environment

• scoring procedure

(yes, the same common sources of measurement error)

true or false: you can try to minimize the effects of the confound variable to reduce error

true

how can you standardize participants to reduce the measurement error?

by deciding what at the inclusion and exclusion criteria for participants before recruiting them

how can you standardize the testing protocol to reduce the measurement error? (4)

by making sure you stay consistent in

• instructions

• treatment

• administration of test/measures

• order of tests/keasures

how can you standardize the environment to reduce the measurement error?

by choosing the best environment for your testing and try to reproduce it in the future

why should you take notes of factors such as time of the day/week/year, temperate, noise level, accessibility…?

they might not be important for your hypothesis, but you can’t control them. they might have an effect later

how can you standardize the scoring to reduce the measurement error?

by making sure the criteria are objective, clear and precise

when should you really do practice runs to become familiar with the scoring procedure?

when there are a lot of raters

measurements can be either […] or […]

• qualitative (categories)

• quantitative (numerical)

what are the scales of measurement? (4)

• nominal: attributes are named

• ordinal: attributes are ordered

• interval: distance between the values is meaningful. 0 means something

• ratio: you can do ratio (× or ÷). 0 means absence of

define “nominal” scale

• values are named

• values are not related in a systematic way

• no mathematical operations, only equality

• ex: what’s your major. someone in psychology isn’t superior to someone in sociology

define “ordinal” scale

• values are named and ordered

• no assumption about the equal differences between categories (we can’t say how far apart each categories are)

• no mathematical operations, only equality and order are possible

  • careful: the equality might be the same for you, but you 4 might not equal someone else’s 4

• ex: spiciness of food. 1 pepper is lower than 2 pepper and 1 pepper taco is as spicy as 1 pepper burrito

→ tells use that scores are different and the direction of difference

define “interval” scale

• values are ordered and can be named

• the distance between each value is the same

• equality, order and mathematical operations like + and -

• ex: temperature

define “ratio” scale

• values are ordered and can be named

• the distance between each value is the same

• equality, order and mathematical operations like +, -, × and ÷

• ex: money (sometimes)

check the correct boxes of this table

nominal, ordinal, interval, or ratio: person’s nationality (1 = Canadian, 2 = American, 3 = Australian, 4 = British)

nominal

nominal, ordinal, interval, or ratio: musical song’s likeability (1 = not much, 5 = a lot)

ordinal: you can rank them, but you don’t know if your 4 is the same as someone else’s 4

nominal, ordinal, interval, or ratio: person’s psychiatric disorder (0 = none, 1= Depression, 2 = anxiety, 3 = schizophrenia)

nominal

nominal, ordinal, interval, or ratio: person’s shoe size (5, 6, 7, 8, 9, 10)

ordinal: you can only order the size

• not interval because 0 would mean absence of person. plus, there is no real standard for shoe size

nominal, ordinal, interval, or ratio: person’s education level (1 = high school, 2 = university-Bachelors, 3 = Masters, 4 = PhD)

ordinal

nominal, ordinal, interval, or ratio: person’s annual income ($ amount)

ratio 0$ means no money

nominal, ordinal, interval, or ratio: person’s annual income (1 = $0k-10k; 2 = $11k-20k; 3 = $21k-30k; 4 = $41k+)

ordinal because it’s ranked

• not interval or ratio because the distance between each group isn’t the same and there is one missing interval

nominal, ordinal, interval, or ratio: person’s height (cm / inches)

ratio because the 0 is meaningless and you can do all mathematical operations

what are the measures of central tendency? (3)

• mean: average of scores

• median: midpoint of the distribution

• mode: most common score

define “central tendency”

statistical measure that identifies the center of the distribution

true or false: you won’t always have a mode

true: especially for smaller groups

true or false: the value of the central tendency must also be a score you obtained

false: not necessarily

identify the mode, media and mean

if you have a ratio or interval scale, which central tendency measure should you use?

mean (median and mode work too)

if you have a ordinal scale, which central tendency measure should you use?

median (mode works too)

• not mean because ordinal is ordered and mean isn’t (just doesn’t make sense)

if you have a nominal scale, which central tendency measure should you use?

mode

why do people use the mode instead of the mean when we ask for the average of something?

• because it’s more perceptually salient

• if you want the average of something, you should precise whether you want the mean, median or mode

explain the “hot hand” view

you expect that someone who experienced success will keep succeeding (based on the modal outcome)

what are some types of measurements? (3)

• self-report: direct measure from the participant

• physiological/neural: indirect, something on top of your skin or skull

• behavioural observation: indirect, viewing something about the participant

explain the self-report measurement

• it’s a direct measure of the participant

• we ask them to report or rate a construct

what’s a pro and a con of self-report?

• con: it’s easy to interpret questions differently, which will affect validity (as if it’s understood incorrectly, you won’t measure the right thing)

• pro: self-knowledge and self-awareness as you can train your participants on how to use the self-report

explain the physiological/neural measurement

• it’s an indirect measure of a construct

• we put detectors on their skin and skull (physiological manifestations)

what’s a pro and a con of physiological measures?

• pro: objective, no objective interpretation

• con: can be expensive or require an unnatural situation

explain a behavioural measurement

• it’s an indirect measure of a construct

• we observe the behaviours of a participant

what’s a pro and con of behavioural measurement?

• pro: easy to access

• con: does the behaviour truly represent the construct

why would you measure your construct with different procedures?

because it would provide more confidence in the validity of the measurements

what are the problems of using multiple measures for one construct? (2)

• statistically speaking, it’s pretty complex

• the two measures might not behave the same way

define “artifact”

no natural feature accidentally introduced into something being observed

why is reactivity a problem occurring in laboratory settings only, and not in field studies?

because on field studies, people are in their environment and might not know that they are being measured