KAAP 400 - Lecture 7

What is the core question behind all research designs?

“If my dependent variable changes, how will I know why it changed?” This helps determine whether the research design can identify what caused the change in the dependent variable.

Why does control matter in research design?

Observing a change is easy, but determining why the change occurred requires careful planning, control, and measurement. Researchers must consider other explanations, such as learning effects, motivation, fatigue, or sleep, before concluding that an intervention caused the change

What are control strategies, and why are they important in research design?

Control strategies reduce alternative explanations and “noise” caused by human variability. They increase confidence that the independent variable (IV) caused the change in the dependent variable (DV)

What is randomization, and why is it important in research design?

Randomization assigns participants to groups by chance rather than choice or convenience. It helps evenly distribute known and unknown confounding variables, reducing bias and increasing confidence that the IV caused changes in the DV. For example, if motivated athletes choose high-intensity training, motivation could become a confounding variable.

What are order effects, and how can they be controlled?

Order effects occur when the order in which conditions or tests are presented affects participants’ responses. They can cause learning effects or testing fatigue. Counterbalancing reduces order effects by randomly assigning participants so some complete the training first and others complete the control condition first.

What is standardization, and why is it important in research design?

Standardization means keeping methods and conditions uniform across participants so the study can be replicated. It reduces unwanted variability and helps isolate the true effect of the intervention. For example, test participants at the same time of day and control pre-test conditions such as sleep and caffeine.

What is blinding, and why is it important in research design?

Blinding means hiding key study information from people involved in the research to reduce bias. Participant knowledge can affect performance, effort, or behavior, while observer knowledge can affect encouragement, feedback, or scoring. Even partial blinding can strengthen study credibility.

What is reliability, and how can you tell if a measurement is reliable?

Reliability is the consistency and reproducibility of a measurement. A reliable measure should give similar results when repeated under the same conditions if the underlying condition has not changed. In the example, scores of 18, 24, and 19 show variability, suggesting the questionnaire may not be highly reliable.

What is test-retest reliability?

Test-retest reliability measures how consistent a test is when given to the same group on two different occasions. A high correlation between scores indicates strong reliability, while a low correlation suggests inconsistency or changes over time. It can be affected by memory, learning, fatigue, or changes in testing conditions. For example, a correlation of 0.85 indicates good test-retest reliability

What is inter-rater reliability?

Inter-rater reliability measures the consistency or agreement between two or more raters assessing the same behavior, performance, or measurement. High inter-rater reliability means raters strongly agree, while low reliability indicates inconsistency or subjectivity. Clear training and scoring guidelines can improve inter-rater reliability.

How can researchers ensure reliability?

Ensure reliability by using standardized procedures, training observers/raters with the same criteria, keeping testing conditions consistent, using clear and objective instructions/scoring guidelines, checking reliability with test-retest or inter-rater reliability, and minimizing outside factors such as fatigue and distractions

What is validity, and how is it different from reliability?

Validity asks whether a measure accurately measures what it is supposed to measure. Unlike reliability, which focuses on consistency, validity focuses on accuracy and appropriateness. A measure can be reliable but not valid—for example, a thermometer that consistently gives the wrong temperature.

What is construct validity?

Construct validity asks whether a measurement actually measures the concept (construct) it is intended to measure. A construct is an abstract idea, while the operational definition explains how it is measured. Convergent validity means related measures show similar results, while discriminant validity means unrelated measures show different results. For example, heart rate alone may not validly measure anxiety because it can also change due to caffeine, physical activity, excitement, or temperature.

What is the difference between internal and external validity?

Internal validity asks how confidently we can say the results were caused by the independent variable rather than confounding factors. External validity asks how well the results can be generalized to other people, settings, or situations. Confounding variables are uncontrolled factors that can influence the dependent variable.

What is internal validity, and what are common threats to it?

Internal validity is how confidently we can say the independent variable caused the change rather than another factor. It is strengthened by randomization and controlled conditions. Common threats include maturation (natural changes over time), learning effects (improvement from repeated exposure), selection bias (pre-existing differences between groups), and attrition (participants dropping out).

What is external validity, and what factors can affect it?

External validity is how well study findings can be generalized to other people, settings, or situations. It asks, “Can these findings be applied beyond this study?” It is influenced by the sample size and participant characteristics, as well as differences in settings and populations. Results from a very specific sample may not apply to other groups, such as different ages or health conditions.

What should you consider when evaluating the external validity of a study?

Consider who the participants were, where the study was conducted, and under what conditions. External validity does not mean findings must apply to everyone; it means researchers should clearly state the limits of generalization and avoid conclusions beyond what the data support.

What should you consider when evaluating the external validity of a study?

Consider who the participants were, where the study was conducted, and under what conditions. External validity does not mean findings must apply to everyone; it means researchers should clearly state the limits of generalization and avoid conclusions beyond what the data support.

How can sample characteristics and treatment–participant interactions threaten external validity?

External validity can be limited when a study uses a small, homogeneous, highly specific, or convenience sample, because findings may not generalize broadly. Treatment effects may also differ across subgroups based on characteristics such as age, experience, activity level, or severity of a condition. Therefore, an intervention that works for one group may not work for another.

What is ecological validity?

Ecological validity is how well study results reflect real-life situations and everyday experiences. Studies in natural, realistic settings generally have higher ecological validity, while highly controlled laboratory settings may be less realistic. It is considered a subset of external validity

What is the Hawthorne Effect, and how can it affect a study?

The Hawthorne Effect occurs when participants change their behavior simply because they know they are being studied. They may work harder, behave differently, or perform better. This can reduce ecological validity because their behavior may not reflect how they normally act in everyday life.

What simple question helps you remember external validity?

“Will X likely cause Y in other people, places, or situations?” External validity focuses on whether study findings can be generalized beyond the original study.

Why does planning research come before experimental design?

Planning ensures the study measures the right constructs consistently and reduces alternative explanations. It helps define the research question, choose an appropriate design, identify and control confounders, standardize data collection, improve reliability and validity, and identify potential problems. A well-randomized study cannot fix unreliable or invalid measurements