Critically Evaluating Research Part 2

4 types of validity to help evaluate the claims

construct, external, internal, and statistical

Construct validity is

does the way they measured or manipulated a variable have a clear connection to the conceptual language used

example of construct validity

some researchers will used the SAT/ACT to talk about “intelligence” NO!!!

To get higher construct validity

need to define the variable better (operationalized the word better)

External Validity is

does the study do things in a way that we can generalize (apply) this findings in the broader population

example of external validity

if you study memory using college students in a lab, can you generalize to older adults in real life?

to get higher external validity

need representative samples

observation study has

high external validity

Internal validity is

how well you can rule out the role of any other variable in relation between 2 variables

example of internal validity

did studying more cause higher test scores, or was it prior knowledge?

to get higher internal validity

require true experiment (get us close to causal claim), best if “within-subject” (repeat with the same participant)

Statistical validity is

are the statistical conclusion accurate?

example of statistical validity

claiming an effect exists when the sample size is tiny

to get higher statistical validity

make sure statistical tests were done and interpreted correctly

Temporal precedence is

the cause needs to happen before the effect

3 basic design types

Experimental, Quasi-Experimental, Observational/Correlational

Experimental design is

does X cause Y?

Experimental design has full control over

who is assigned to the levels of the variable of interest and over everything else (settings)

Experimental design leads to

strong causal conclusion, internal validity, and establish X comes before Y

example of experimental design

randomly assigning participants to sleep deprivation vs. normal sleep and measure memory

Experimental design needs 3 criteria: 1)

Covariance → X and Y must show a relationship

Experimental design needs 3 criteria: 2)

Temporal precedence → X must come before Y

Experimental design needs 3 criteria: 3)

Internal validity → have rule out that any variables other than X is producing the differences in Y

Quasi-Experimental design:

is X related to Y when random assignment isn’t possible?

Quasi-Experimental design has

some to full control over who is assigned to the levels of the variable of interest

Quasi-Experimental design has little control over

everything else (setting)

Quasi-Experimental design is the best way to balance

external and internal validity (real-world setting when random assignment isn’t possible but still have some control)

Quasi-Experimental design can establish

temporal precedence (X comes before Y)

example of quasi-experimental design

A school introduces a new math teaching method in one 10th-grade class but not in another

• manipulated variable: new vs. traditional teaching

• groups: pre-existing classes (no random assignment)

Observational/Correlation design:

are X and Y related?

Observational/Correlation design has

no control over who is assigned to levels of the variable of interest (no manipulation of the variable) and everything else (setting)

Observational/Correlation design show good _____ but poor _____

external validity, internal validity

Observational/Correlation design relies on

statistical control (which is difficult to do)

Temporal precedence can be established in Observational/Correlation design only if

1) repeated measures (sometimes)