Lecture 7: research strategies and validity (part 2)

what are the threats to external validity? (3)

• generalizing across participants: can results be generalized to people who differ from the participants

• generalizing across studies: can results be generalized to other procedures (manipulation, IV)

• generalizing across features of the measures: can results be generalized to other methods measuring the study (measure, DV)

what”s the difference between “internal validity” and “external validity”?

• internal: extent to which you can confirm that the changes in X causes the changes in Y

• external: extent to which your results can be generalized to other settings and populations

define the first threat to external validity, “generalizing across participants”

to what extent can results be generalized to people who differ from the participants

what are some characteristics that threaten the generalization across participants? (5)

• selection bias: favouring the selection of some people

• university students: can have better memory

• volunteer bias: volunteers might no be representative of the population

• participant characteristics: your sample might contain more of a certain characteristic

• cross-species generalization: what applies to animals might not apply to humans

define “selection bias”

favouring the selection of certain individuals, sometimes out of convenience

→ threaten external validity

how can the fact that a sample has university students might threaten external validity?

university students are shown to have a less formulated sense of self, a tendency to comply with authority, less stable peer relationships and higher intelligence than non college people

define “volunteer bias”

volunteers aren’t perfectly representative of the population

how confident are we that these characteristics represent volunteers?

what are some demographic characteristics that tend to threaten external validity? (4)

• age

• gender

• socioeconomic status

• education level

define “cross-species generalization”

what can be applied to animals can’t necessarily be applied to humans

define the second threat to external validity, “generalizing across studies (design features)”

• to what extent can your results be generalized to other procedures

• characteristics that are unique to a procure or experimenters limits generalizability to situations with other procedures or experimenters

what are some elements that limit generalization across studies? (3)

• novelty effect: in a new situation, you act differently

• multiple treatment interference: if you give multiple treatments, there might be interference from the previous treatment

• experimenter characteristics: different experimenters can give you different results

define “novelty effect”

• new situations can elicit artificial responses

• (or if your stimuli/task isn’t realistic, you might not measure a realistic behaviour)

define “multiple treatment interference”

if you give multiple treatments, there might be interference from the previous treatment

what are the carry-over effects that causes interference between treatments? (2)

• fatigue effect: you get tired quickly if a task requires cognitive capacity

• practice effect: you tend to get better with practice

define “experimenter characteristics”

different experimenters can give you different results

define the third threat to external validity, “generalizing across features of measurements”

• to what extend can research results be generalized to other methods of measurement in the study

• if your procedure is too specific, then your results might not be generalized to study with different procedures

what are some elements that limit generalization across measurements? (3)

• sensitization: process of measurement alters your behaviour

• generalization across response measures: will the results be the same if you measure the variable differently

• time of measurement: will the results be the same if you measure the variable at another time

define “sensitization”

process of measuring alters your behaviour during the study because you know you are being measured

what could sensitize participants even more? (2)

• pre-test measures (AKA pre-test sensitization)

• self-monitoring

→ they both affect self-awareness and might change their behaviour because of it

define “generalization across other measures”

we don’t know if we would have obtained the same results if the variable was measured another way

define “time of measurement”

we don’t know if we would have obtained the same results if the variables were measured at another time of the day

what are the categories of confounding variables that threaten internal validity? (3)

• environmental variables: characteristics in the environment that differ between conditions

• individual differences: characteristics that differ from one person to another

• time-related variables: time lapse between measuring

define the first threat to internal validity, “environmental variables”

characteristics in the environment might differ between the treatment conditions

define the second threat to internal validity, “individual differences”

characteristics that differ from one person to another

define the third threat to internal validity, “time-related variables”

• when you test the same participant multiple times, the time lapse can cause some changes

• the changes can come from the environment or from the participant

what are some time-related variables? (4)

• history: outside event that changes and affects the participants’ scores

• maturation: participant matures physically or psychologically between treatments

• instrumentation: change in measuring instruments over the course of the study

• testing effects: reactive, fatigue, carry-over

define history and explain how it is a time-related variable that affects internal validity

• an outside event that changes over time and that affect participants’ scores

• it must influence at least one condition and enough participants to affect the group average

define maturation and explain how it is a time-related variable that affects internal validity

• when a participant changes physically or psychologically between treatments

• problem during long-term studies

• problems with young (grow) and old (health and cognition)

define instrumentation and explain how it is a time-related variable that affects internal validity

• when there are changes in the measuring instrument over the course of the study

• you can get different scores because of the instrument (because it got used and less sensitive)

define the “practice effect”

• people get better, faster and more accurate with repeated testing

• meaning that practice improves from trial and week

define “power law of learning”

you get better quickly at the beginning, but then it kinda plateau

define the “fatigue effect”

• some cognitive capacities are limited, mental fatigue results in reduced accuracy and slower reactions

• fatigue can occur quickly within an experiment

define the “carry-over effect”

when a previously experienced condition affects participants’ responses to a current condition (because of the order)

define “experimenter bias” (description, threats to external and internal validity)

• results are influenced by the experimenter

• external: results specific to that experimenter and may not generalize

• internal: results are caused by the experimenter and not the IV

define “demand characteristics” (description, threats to external and internal validity)

• cues that tell the participant what the study is about which influences their behaviour

• external: results specific to that experimental design and might not generalize

• internal: results are caused by demand characteristics and not by the IV

define “participant reactivity” (description, threats to external and internal validity)

• participants’s know they are being measured which influences their behaviour

• external: results are specific to that participants and may not generalize to the population

• internal: participants’s role may explain the results and not the IV

what are the artifacts that threaten both internal and external validity? (3)

• experimenter bias: results are influenced by experimenter

• demand characteristics: cue that tell the participants what the study is about (which might influence their behaviour)

• participant reactivity: participant know they are being observed so they change their behaviour

internal or external validity: Researchers had a single group of university students take notes for a class lecture by hand and at a later date the group took notes for a different lecture on laptops. They were tested afterwards on their memory for the lecture contents. Tests following the handwritten note condition performed better. The students were recruited from a single lecture class at a university. This might have limited their […] validity.

internal: the order of note could have changed the results or maybe it’s because of the time elapsed

internal or external validity: Researchers had a single group of university students take notes for a class lecture by hand and at a later date the group took notes for a different lecture on laptops. They were tested afterwards on their memory for the lecture contents. Tests following the handwritten note condition performed better. The lecture content was a TED talk for the handwritten condition and a professor's lecture for the laptop condition. This might have limited their […] validity.

external: are TED talks the same as lecture? we can’t generalize that

true or false: internal and external validities are positively correlated (as one goes up, so does the other)

false: as one increases, the other decrease

• if external is high, the internal is low because we didn’t control the environment enough

  • high external = good world generalization

  • low internal = not sure if x caused y

• if internal is high, then external is low because we controlled the environment too much

  • high internal = confident that x caused y

  • low external = not generalizable to the world

why do researchers sometimes exaggerate the differences between variables?

• so that the scores obtained in one treatment are noticeably different from the other treatment

• the goal of research is mainly to show a relationship between variables

• meaning that we maximizes the differences for one variable to increase the likelihood of revealing the difference with the other variable

define “artifact”

external factor that may influence of distort measurements

what are the “fact of life” concerning the validity and quality of a research study? (2)

• studies differ in terms of validity

• being aware of threats to validly help you critically evaluate a research study