Introduction To Psychological Experimentation - Concept List

Research question vs hypothesis vs prediction

Research question: a broad question about the world, motivated by a real problem (e.g. does reading Lord of the Rings make Miles more like Gandalf?). Hypothesis: a construct-level claim grounded in theory that can be tested many ways (e.g. reading about a fictional character increases identification with that character). Prediction: the specific expected result on the measure you chose (e.g. faster RTs for congruent than incongruent IAT pairings). Order: question → hypothesis → operationalisation → prediction. Keeping them distinct protects against HARKing (hypothesising after results are known).

Operationalisation

Turning a concept into something measurable, i.e. choosing the tool that turns it into a number. One hypothesis can be operationalised many ways (IAT, questionnaire, fMRI, observation). E.g. using the IAT to measure Miles's identification with wizards. It is the key lever for limiting concept overreach.

Dependent vs independent variables

Independent variable (IV): the thing the researcher changes or manipulates (e.g. IAT congruency, which book was read). Dependent variable (DV): the thing that is measured or counted (e.g. reaction time).

Manipulation vs correlation designs

Manipulation: the researcher changes an IV and measures the effect on a DV, which allows a cause-and-effect inference. Correlation: two measured variables are compared to see whether they move together. You can't infer causation from correlation, although a causal link could still exist.

Within vs between vs mixed designs

Within-subjects (repeated measures): the same participants do every condition. It has more power and needs fewer participants, but risks order effects and demand characteristics. Between-subjects (independent groups): each participant does only one condition. It has more noise and needs more participants. Mixed: at least one within IV and one between IV (e.g. congruency within, book read between).

One-way vs factorial designs

One-way: one IV, which can have 2 or more levels (e.g. congruent vs incongruent, analysed with a t-test). Factorial: 2 or more IVs, named by the number of levels of each (2×2, 2×3, etc.). It shows how the IVs interact, which makes it very powerful. Levels are also called conditions.

Quasi-experimental designs

Designs where at least one IV cannot be randomly allocated because it is a fixed participant characteristic (e.g. sex at birth, young vs old). Weakness: the groups may differ in other ways (e.g. love of Lord of the Rings), which creates alternative explanations. E.g. the odour study was a 2×7 quasi-between design.

Control conditions

A baseline condition for comparison, so an effect can be attributed to the manipulation (e.g. Juliet reading Pride and Prejudice instead of Lord of the Rings). The choice of control decides what effect you are testing: a Harry Potter control would test whether Gandalf specifically matters. The odour study failed because it didn't compare against its blank-pad control.

Extraneous variables vs confounds

Extraneous variable: any variable other than the IV that could affect the DV (background noise, distraction). It is acceptable if spread equally across conditions, but it adds noise. Confound: an extraneous variable that varies systematically with the IV, so you can't tell which one caused the effect (e.g. a wizard hat left only in the LOTR condition, or one condition run during a heatwave). Confounds destroy internal validity.

Sampling bias

When the way participants are recruited makes a sample, or one condition, systematically different. E.g. recruiting the LOTR condition outside Comic-Con means those participants already love wizardry, which makes it a confound. Sampling bias also limits generalisation (e.g. results from free divers applied to the general public).

Sample characteristics

Descriptive information about participants (age, sex, gender, other relevant traits). It is collected to check that conditions are broadly similar and to describe who the results apply to.

Attention checks

Items within an experiment that check participants are actually paying attention to the task. They matter especially online, where people get bored, watch Netflix or make tea.

Manipulation checks

Checks that participants actually experienced the manipulation as intended. E.g. the odour study checked that sniffers could actually smell.

Counterbalancing

Varying the order of conditions across participants so order effects cancel out in within-subjects designs. E.g. half do congruent then incongruent IAT blocks, half do the reverse, and the results are averaged.

Randomisation

Randomising aspects of the experiment (e.g. trial or condition order) so extraneous variables are spread evenly and don't become confounds. You can't remove noise, but you can control it through randomisation.

Random allocation(assignment)

Randomly assigning participants to conditions in between-subjects designs, so groups are broadly similar (age, interests, love of wizardry). It is not possible in quasi-experimental designs.

Inferential overreach

Claiming more than the evidence supports when moving from evidence to claim (e.g. "reading Lord of the Rings turns Miles into a wizard", or "brain rot" headlines). There are 5 types: causation, concept, relevance, explanation and certainty. The goal is a bounded claim that stays close to the evidence.

Construct vs internal vs external validity

Construct: does the measure actually capture the concept? (Does the IAT measure wizardry?) It guards against concept overreach; the lever is operationalisation. Internal: does the IV actually cause the change in the DV, with no confounds? It guards against causation overreach; the lever is control and design. External: do the results generalise to other people, settings and times (including ecological validity)? It guards against relevance overreach; the levers are sampling and realistic measures.

Reliability

The consistency of a measure: it gives similar results over time (test-retest) and its items hang together (internal consistency, e.g. Cronbach's alpha). It relates to both concept and relevance overreach.

Convergence

When multiple studies using different methods or samples point in the same direction. No single study proves a hypothesis, so results must be integrated with existing research. It guards against certainty overreach ("standing on the shoulders of your peers").

Underdetermination

When the same outcome can be explained by more than one hypothesis. E.g. caffeine improving word recall could reflect memory, attention, perception, or just the comfort of a warm drink. It guards against explanation overreach: discuss alternative explanations.

Statistical evidence

Using statistical tests (e.g. t-tests, p-values) to judge whether a result is unlikely to be due to chance. Significance is not proof and still needs interpretation. E.g. the odour study had p < .00001 but compared the wrong conditions.

Causal Overreach

Claiming that manipulation X caused outcome Y when the design can't support it. This is a failure of internal validity. Limited by good control: control conditions, random allocation, counterbalancing, and checking for confounds. E.g. the odour study claimed older women's odour reduced depression, but the blank-pad control showed people got less depressed anyway.

Concept Overreach

Claiming experiment X measured concept Z when the measure might not capture it. This is a failure of construct validity. Limited by careful operationalisation. E.g. claiming a 200ms IAT difference is "pure wizardness", or a study of brain networks claiming adults "think like teenagers". The odour study also used a trait scale to measure an in-the-moment (state) mood change.

Relevance Overreach

Claiming outcome Y generalises beyond the experiment to other populations, settings or times. This is a failure of external validity (including ecological validity). Limited by sampling and realistic measures. E.g. a brain-benefits study of 17 free divers applied to the general public. Moss-Racusin et al. guarded against it by making the CV-rating task realistic for professors.

Explanation Overreach

Claiming your hypothesis uniquely explains the outcome when other hypotheses could explain it too. This is a problem of underdetermination. Limited by having good hypotheses and discussing alternative explanations. E.g. claiming caffeine improves memory when it might improve attention or perception instead.

Certainty Overreach

Claiming one outcome proves your hypothesis. No single experiment fully answers a research question. Limited by integrating results with existing research (convergence) and acknowledging that other studies, with other methods, are needed. Avoid hubris: "standing on the shoulders of your peers".

Alternative Explanations

Other plausible reasons, besides your hypothesis, for why a result occurred. Examples: confounds, a different mechanism (attention rather than memory), demand characteristics, order effects, or sample differences. Discussing them in the discussion section shows good scientific reasoning and prevents explanation overreach. Media reports often drop them.

Block Designs

Grouping together trials that share the same condition into "blocks" (e.g. in the IAT, all congruent trials are blocked together and all incongruent trials are blocked together). Because the IAT is a one-way within-subjects design (IV = congruency, 2 conditions), everyone sees both blocks, so block ORDER is counterbalanced: 50% do congruent first, 50% incongruent first, to control for order effects. Order is just a control, not analysed (it made no difference to IAT size).

P-values vs Effect Sizes

P-value = tells you whether a difference is statistically significant. If p < .05, there's less than a 5% chance of a difference this large if there were really no difference, so we reject the null and accept the alternative (e.g. the 106 ms congruent vs incongruent difference in the IAT, 814 ms vs 921 ms). Effect size (e.g. Cohen's d) = tells you HOW BIG the difference is, independent of sample size. A large sample can make a tiny effect significant, so APA results report both.