Lecture 8 - Reading Scientific Papers

Independent Variable

Manipulated or controlled by experimenter (e.g., whether adults put in effort vs no effort)

• In experiments, the IV = the conditions in the study

• Almost always depicted on the x-axis

Dependent Variable

Behavioral or survey response that’s the outcome of interest (e.g., whether the infant put in effort)

• Studies often have multiple outcome variables

• Almost always depicted on the y-axis

Control variable

variable that is deliberately kept fixed or constant, in order to ensure that only differences in the IV cause the DV

• Ex: the room the infant was in, the task completed by the adult, the adult’s gender, etc.)

Covariance

do the results show that the causal variable is related to the outcome variable? Are distinct levels of the IV associated with different levels of the DV? In other words, does a change in the IV produce a change in the DV?

• correlational studies only establish covariance, occasionally

Temporal precedence

does the study design ensure that the causal variable comes before the outcome variable in time?

Internal validity

does the study rule out alternative explanations for the results? Does experiment design have confounds? Experiments with confounds don’t meet this criterion

Control group

given “nothing,” while the treatment group is given something

• Ex: I want to test whether taking Data Literacy for Psychology makes you happier

  • Some students take DL, others no course: not practical and not well controlled (has confounds)

Comparison group

is given “something else” than the treatment group

• Ex: I want to test whether taking Data Literacy for Psychology makes you happier

  • Some students take DL, others an Advanced Stats course: comparison group (has confounds)

Confounds

Variables that are not the IV but which vary across conditions

• You must design your experiment to avoid confounds. Controlling for everything but the hypothesized causal variable is the best safeguard for internal validity

• Confounds threaten an experiment’s internal validity because confounds serve as alternate explanations

Confounds are systematic differences

• Confounds imply systematic variability

• Participants in different conditions will differ in random ways

  • Some DL students got coffee before class, but some Advanced Stats students may have also 

  • Some DL students come in late, but perhaps true for Advanced Stats 

• Random variability always occurs due to random differences in sampling, but systematic variability between conditions is part of the study design

  • Somewhat parallel to the difference between random/sampling error vs systematic error 

Selection effects

Systematic differences between participants in one condition vs the other

• At NYU, students are not randomly assigned to DL or Advanced Stats; they choose the course themselves. Maybe students who are more adventurous choose DL, and maybe adventurous students are also happier. Being adventurous is a confound. 

• Let’s say in a hypothetical study for treating autism, some parents insisted that their children be in the new intensive-treatment groups rather than the treatment-as-usual group

• Can this experiment determine whether the improvement in the intensive group was caused by the treatment? 

  • No, there is a confound: some parents who were more motivated chose the intensive treatment, so we can’t be sure if improvement is caused by the treatment or higher motivation

Random assignment

Participants are randomly assigned to one of the study’s conditions. This ensure that the two groups are equivalent

Matched assignment

the two groups of participants are matches as much as possible on various dimensions, making equivalent groups

• Ex: if researchers were trying out a form of weight loss drug, the participants would need to be matched to make sure they were all the same weight, height, build and had similar diets

• How would you do matches assignment to Data Literacy vs Advanced Stats?

  • You could match by major, gpa, prior stats experience, etc.

• More cautious (and laborious) than random assignment

  • BUT still risks confounds (when not combined with random assignment)

Between-subjects/independent-groups design

Separate groups are presented with different levels of the independent variable

• Some students take DL, others take Advanced Statistics

• Some children get treatment A, others get treatment B, and even others get C

• Some takes notes on the laptop, others with their hand

Within-groups/within-subjects design

the same individual is presented with different levels of the IV. each participant is their “own control” or comparison

• Big advantage: removing concerns about individual variability. We can ask: how do the same individuals respond differently to different treatments?

• Fewer participants but longer experiments

• Most common type of within-subjects design is the repeated-measures design, wherein participants are measured on a DV more than once, after exposure to each level of the IV

• Ex: The same students take DL one semester, and Advanced Stats the other semester

  • The children get all three treatments - A, B, and C - one after the other

  • The same students take notes on the laptop one day and with their hand the next day

Posttest-only

the DV is measured just once post-manipulation

Pretest-posttest

the DV is measured once pre and once post the manipulation

• Given random assignment, the pre-measure for both groups should be equal

Why is a within-subjects design still an experiment, without random assignment?

• Random assignment prevents selection effects and person confounds

  • I.e. fundamental differences between groups

• In within-groups designs, all participants are in all conditions

  • There are no selection effects or person confounds

• What confound might exist in within-subjects designs?

  • Order effects - make sure you randomize the order of treatments to prevent these

Pretest-posttest vs repeated measures

• In a pretest/posttest design, participants see only one level of the IV

• In a repeated-measures design, participants are exposed to all levels of the IV (even if order is randomized)

Construct validity

how well is the IV manipulated and the DV manipulated?

• If one wants to measure happiness, asking participants’ job satisfaction is a bad measure, but asking how happy they feel is a better measure

• If I want to study the effect of exposure to humor, a bad joke is a bad manipulation (i.e., it doesn’t manipulate humor well). Manipulation checks (or questions that measure humor post exposure to the manipulation) or pilot studies can help

Internal validity

are there alternative explanations for the results? Are there confounds?

External validity

to whom or what can the causal claim generalize?

• To whom? WEIRD studies may not generalize to non-WEIRD samples. Often requires follow-up studies. Random sampling from a population and representative samples address this issue

• To what? If I’m interested in studying the effect of humor on mood, I may have you read a funny joke and then measure your mood. Will the findings generalize to other jokes? You ideally want to test with multiple jokes

Statistical validity

• How large is the effect? Is the effect large enough to matter? If a researcher tests the effect of a flu medicine, the effect should be large enough to be of value in the real world

• How precise is the estimate? Large sample produce more precise estimates (recall that large sample have lower margin of error)

How should we read a paper?

• Papers are full of very specific details that are useful for researchers who want to replicate/re-run the same study, but more of the details are not necessary for understanding/critiquing the study

• The QDAFI method can be helpful for quickly and efficiently reading papers

  • This method exploits the canonical structure of papers and helps focus on the gist

  • Over time, we remember only papers’ essence anyway

  • Focusing on the paper’s essence enables better understanding

Q (Question)

• Each paper starts w/ a question that the authors set out to answer. State this question explicitly, in your own words, so we can gauge understanding

• Single sentence

• Q: Does taking class notes by hand yield better academic performance than taking them on a laptop? (Mueller & Oppenheimer, 2014)

D (Do)

• What did the authors do to answer the question? (Methods)

• What did they manipulate (IV)? What did they measure (DV)?

  • If they did a lot of stuff, what was the most important such IV/DV pairing regarding the question?

• Single (or at most 2) sentence

• D: Participants were asked to either take notes on a laptop or by hand while watching TED talks. They were then asked to answer both factual and conceptual questions about the material presented in the talks (Mueller & Oppenheimer, 2014)

A (Authors’ rationale)

• Links the research question (Q) to the methods that the authors used (D). how would the method help answer the question? Or, given the theory, what’s the expected finding? Usually found in the paper’s Introduction section

  • E.g., “If [DV] were found to depend on [IV] in [some way], then this would demonstrate that [answer to Q], because…”

  • E.g., “If the answer to [Q] were…., then we would expect [DV] to depend on [IV] in [this way] because…”

  • E.g., “It is believed that… [hypothesis]. If this were so, then we would find [dependence between IV and DV].”

• One or two sentences

• A: It is believed that using laptops to take notes yields poor performance because of distractions. However, even without distractions, if laptop use leads to shallower processing, it could detrimentally impact performance on conceptual questions

F (Finding)

• What is the finding set up by D (method) and A (rationale)?

• Single sentence

• F: students who types notes on a computer performed worse on conceptual but not factual questions compared to those who wrote them by hand

I (Interpretation/issues/own interpretation)

• How did the authors interpret their findings or what conclusions did they make about the original question?

• F: Taking notes on a laptop negatively affects performance in response to conceptual questions compared to taking notes by hand, which could be due to the fact that taking notes on a laptop seems to encourage shallow processing, such as copying the material verbatim

• What’s your take on their interpretation? Are there confounds? You could think about the 4 types of validity

• One sentence if the interpretation is legit, and two if it is not

  • If it is not, the second sentence could explain the problem you identified and the alternate interpretation