RESEARCH 2. EXPERIMENTS

Bivariate Correlations

Associations described between exactly two variables. The data of the association can be plotted as a bar graph or a scatter plot. The reported result may be a correlation coefficient or a difference between means.

Correlation Coefficient R

A variable indicating the direction and strength of an association.

Point Estimate

The variable that is the result of statistical analysis. In an association claim, it can take on the form of a correlation coefficient or a difference between means.

Effect Size

The strength of an association. They can indicate the importance of a result, as all else being equal, larger ones are more important. Ones of .30 (or -.30) signify a fairly strong relationship, while .20 (or -.20) signifies a moderately strong relationship and anything near or below .10 (or -.10) signifies a weak relationship. However, ones of around .40 (or -.40) are unusually large in psychology.

How can a small effect size end up becoming important?

Compounding, which can lead to large consequences (a small relationship between two factors can eventually lead to a significant strengthening of that relationship).

Confidence Interval

A range that signifies the margin of error of the point estimate and how precise it is. It is designed to include the true population value at a high proportion of the time (usually 95%). If it does not include zero, the relationship is statistically significant, but if it does, the relationship is not statistically significant. Smaller sample sizes result in wider intervals and less precise estimates, while larger sample sizes result in narrower intervals and more precise estimates.

Outlier

An extreme score originating from a single case that stands out from the rest of the data and can make an association appear more stronger or weaker than it actually is. It can drastically affect associations if not accounted for. They are usually more problematic if they display extreme values on both variables, resulting in a deceptively strong association, and/or if the study has a small sample size.

Restriction of Range

When a restricted range of scores within a certain interval of values is displayed in one of the variables. It can make an association appear weaker than it actually is.

Curvilinear Association

A correlation coefficient that is close to zero as the relationship between variables does not follow the path of a straight line. They can hide the relationship between values or make it appear much weaker than it actually is, as r values will not describe the data well for them.

Replicability

The condition in which other studies testing the same research question yield similar results. If a result has been replicated, we can be more confident about the association.

Is it necessary to interrogate internal validity for association claims?

No, but it is important to try not to be tempted to make causal inferences when investigating association claims.

Spurious Correlation

A correlation that exists only because of a third variable. Several other variables are possible to identify that could potentially explain a bivariate association, although the additional variable must erase any overlap and separate scores based on how it affects the other two variables.

Moderator

A variable that influences the strength and/or direction of the relationship between two other variables. They can inform external validity, revealing the different conditions or groups within which the association holds true or is generally stronger or weaker.

From a theory, how can validity be ensured before starting an experiment?

Construct validity can be established through operationalization, statistical validity can be ensured through checking association direction, effect size, spuriousness, and other factors, and internal validity can be established through setting up comparison groups and accounting for confounds.

Selection Effect

A threat to internal validity where the kinds of participants in one condition systematically differ from those in the other. It no longer ensures that only one element is changed between the two comparison groups and as a result, a causal claim cannot be made regarding the independent and dependent variables. Random assignment helps avoid selection effects.

Matched-Groups Design

A type of experiment that matches participants on a variable that is believed might affect the dependent variable. They create pairs of participants with matching scores on that variable and randomly assign each participant from the pairs to two separate experimental conditions. It is an attempt to create equivalent groups with the presence of a small sample size and characteristics present that cause individual differences.

Control Variables

Variables that remain constant throughout the experiment. They are essential for establishing internal validity because they remove potential confounds and ensure that the independent-dependent variable relationship is the only one being observed.

Design Confound

An experimenter’s mistake in designing an independent variable such that a second variable happens to vary systematically with the intended independent variable. They are major threats to internal validity There will always be some variability across groups in experiments, but variables can only become confounded when their levels vary systematically across levels of the independent variable.

Independent-Groups Experiment

A type of experiment where different participants experience different levels of the independent variable. They can either take the form of posttest-only experiments or pretest/posttest experiments.

Posttest-Only Experiment

A type of independent-groups experiment where the dependent variable is measured only once after exposure to the independent variable. Use of this design is more effective than is the pretest/posttest design.

Pretest/Posttest Experiment

A type of independent-groups experiment where the dependent variable is measured twice, once before and once after exposure to the independent variable. In some instances, use of this design can be problematic, such as if the dependent variable causes fatigue or familiarity effects upon initial exposure. The design can be susceptible to many threats to internal validity.

Within-Groups Experiment

A type of experiment where all the same participants experience different levels of the independent variable. This type of design essentially eliminates selection effects and any potential unsystematic variability. However, there can be potential issues regarding demand characteristics, as participants can act in different ways based on knowledge gained upon exposure to all levels of the independent variable.

Repeated Measures Experiment:

A type of within-groups experiment where participants are exposed to levels of the independent variable sequentially. This design can be susceptible to order effects.

Order Effects

A design confound and a threat to internal validity where exposure to one level of the independent variable can influence responses to subsequent levels of the independent variable in a repeated measures experiment. Differences in the dependent variable can be explained by the sequence in which levels of the independent variable may be experienced. It can come in the form of practice/fatigue effects and carryover effects and it can be prevented through counterbalancing.

Practice/Fatigue Effects

An order effect where participants may show better performance as a task continues or get tired/bored around the end.

Carryover Effects

An order effect where some form of contamination occurs between conditions, with one effect carrying over to the other.

Counterbalancing

A technique used to avoid order effects in experiments through the presentation of conditions in varying orders. It can either be full or partial.

Full Counterbalancing

Counterbalancing where all potential sequential orders of independent variable levels are used.

Partial Counterbalancing

Counterbalancing where not all potential sequential orders of independent variable levels are used. A random order is still used for each participant.

Concurrent Measures Experiment

A type of experiment where participants are exposed to levels of the independent variable simultaneously. Use of this design is preferred over the repeated-measures design.

Manipulation Check

An additional measure researchers can insert into an experiment in order to determine whether or not that their experimental manipulation worked. When an independent variable relates to a relatively abstract construct, they can be used to confirm that the groups actually differ the way they are supposed to (such as asking a question pertaining to how the independent variable was perceived).

Pilot Study

A simple, preliminary study with a separate group of participants from the main study that is completed to confirm the effectiveness of experimental manipulation.

Observer Bias

A threat to internal validity where the expectations of the researchers influence how they interpret the results of their study. Unlike most threats to internal validity, it is likely to occur even with clear comparison groups. Performing a double-blind study or utilizing a masked design can prevent this.

Demand Characteristics

A threat to internal validity where participants guess what the study is about and change their behavior in the expected direction as a result. Unlike most threats to internal validity, they are likely to occur even with clear comparison groups. They have a potential to occur in within-groups experiments, as participants can act in different ways based on knowledge gained upon exposure to all levels of the independent variable. Performing a double-blind study or utilizing a masked design can prevent this.

Placebo Effects

A threat to internal validity where improvement is observed with the administration of a treatment but only because they believe that the treatment they are receiving is valid or effective.

Maturation Threat

A threat to internal validity consisting of a change in behavior that emerges more or less spontaneously over time, such as development, improvements in health, etc. Comparison groups help reveal whether there is an effect of an independent variable beyond or in addition to any maturation effects.

History Threat

A threat to internal validity consisting of a “historical” or external factor, usually an event (seasonal changes, political events, activities, etc.), that systematically affects a majority of the treatment group around the same time as the treatment itself. Comparison groups help reveal whether there is an effect of an independent variable beyond or in addition to any history effects.

Regression to the Mean

A phenomenon where because a group average is unusually extreme when measured at a previous time, it is likely to become less extreme, or closer to the group’s true mean, when measured afterwards. It is only a threat to internal validity when a pretest/posttest design is used and the score is extreme according upon pre-test measurements. Comparison groups and careful inspection of result patterns help rule out the possibility of this.

Attrition

A phenomenon where a certain kind of participant drops out of the study. It is only a threat when the attrition is systematic.

Testing Threat

A threat to internal validity consisting of a change in the participants due to them undergoing measurement of the dependent variable more than once in pretest/posttest designs. Comparison groups, utilizing only posttest only designs, and using different versions of pretest/posttest designs help prevent testing threats.

Instrumentation Threat

A threat to internal validity consisting of changes in the instruments of measurement utilized in the experiment over time. In observational studies, the instruments are people. Calibrating tests if two different tests are used, utilizing clear coding manuals, counterbalancing versions of the test, and utilizing only posttest only designs help prevent instrumentation threats.

Selection-History Threat

A combined threat involving selection and history threats where an outside event systematically affects participants at one level of the independent variable.

Selection-Attrition Threat:

A combined threat involving selection and attrition threats where participants in only one level of the independent variable experience attrition.

What is a common trend between many of the threats to internal validity?

They are likely to occur in one-group pretest/posttest designs and can often be prevented with the addition of a comparison group.

Null Effect

When the independent variable is expected to make a significant difference in the dependent variable in a study when in fact there is not. If the independent variable does not make a difference, some possibilities may be that there are not enough between-groups differences or too much unsystematic, or within-group, variability. Null effects can be just as interesting as results showing group differences but they are published less often and are less likely to be reported in popular media than other results.

For null effects, what could a lack of between-groups differences be caused by?

Weak manipulations, insensitive measures, ceiling and floor effects, and design confounds that act in reverse.

Weak Manipulations

A cause of a lack of between-groups differences. When the difference between levels of the independent variable is too small to be meaningful. It can lead to ceiling and floor effects. It can be revealed through a manipulation check.

Insensitive Measures

A cause of a lack of between-groups differences. When the operationalization of the dependent variable does not have enough sensitivity to detect a difference between levels of the independent variable. It is best to use detailed, quantitative increments in dependent variable measures to avoid this.

Ceiling Effect

A cause of a lack of between-groups differences. When all the scores of participants are squeezed together at the top end of a dependent variable scale. It can be the result of a problematic independent and dependent variable. It can be revealed through a manipulation check.

Floor Effect

A cause of a lack of between-groups differences. When all the scores of participants are squeezed together at the bottom end of a dependent variable scale. It can be the result of a problematic independent and dependent variable. It can be revealed through a manipulation check.

Reverse Design Confound

A cause of a lack of between-groups differences. A design confound that hides the true effect of the independent variable on the dependent variable with a null effect.

For null effects, what could an excess of within-groups variability be caused by?

Measurement errors, individual differences, and situation noise. Unsystematic variability is not a problem for internal validity but it could make it harder to find a true difference between conditions.

Measurement Error

A cause of an excess of within-groups variability. An extraneous factor attributed to humans or measurement instruments that can randomly inflate or deflate a person’s score on the dependent variable. A group’s mean on a dependent variable will reflect the true mean plus or minus any random error. When the distortion of errors are random, they cancel each other out and do not affect the group mean. However, measurement errors can result in scores that look more variable and spread out, resulting in more difficulty in detecting differences between groups. Using more reliable and precise measurements and measuring more instances/participants can prevent this.

Individual Differences

A cause of an excess of within-groups variability. Differences across participants that add variability in dependent variable scores. Changing the design, such as using within-groups designs instead of independent groups designs, and adding more participants can help alleviate this.

Situation Noise

Any external distractions that could cause variability within groups obscuring between-groups differences. It can be minimized by controlling the surroundings of an experiment.

Power

The likelihood that a study will yield a statistically significant result when the independent variable really has an effect. It leads to more precise estimates and can be improved with within-groups designs, strong independent variable manipulation, large sample sizes, and less within-groups variability.