Research Design and Analysis Exam 2

Vocabulary practice flashcards covering measurement concepts (reliability, validity, scales of measurement, observer/experimenter effects) and complex factorial experimental designs based on the review notes.

Test-retest reliability

The degree to which a test gives similar results when administered to the same people at two different times, assuming the measured characteristic remains stable.

Item-total correlation

A measurement of how strongly an individual question on a test relates to the overall test score, evaluating whether that item fits with the rest of the test.

Internal consistency reliability

The extent to which all items or questions on a test consistently measure the same underlying construct. Cronbach’s Alpha: all the possible split-half reliabilities

Interrater reliability

The extent to which two or more independent observers or raters agree and provide consistent ratings of the same behavior. Two psychologists independently rate a child's aggression and agree.

Measurement error

The difference between an observed measurement and the true value. A scale reads 155 pounds when the actual weight is 150 pounds.

Cronbach’s alpha

A statistic representing all possible split-half reliabilities that measures internal consistency among test items, where higher values indicate items work well together. It tells us how well the questions on a test or survey work together to measure the same thing. Imagine a psychologist creates a survey to measure anxiety using five questions: I feel nervous frequently. I worry about things I cannot control. I have trouble relaxing. I often feel overwhelmed. I feel anxious in social situations.

• High Cronbach’s alpha: The questions are closely related and tend to produce consistent responses.

• Low Cronbach’s alpha: The questions don't relate well to one another and may be measuring different things.

Cohen’s kappa

A statistical measure of agreement between two raters assigning categorical ratings that corrects for agreement expected by chance.

Face validity

The extent to which a test appears or looks like it measures what it claims to measure.

A math exam looks like a math exam because it contains math problems.

Content validity

The extent to which a test adequately covers all important parts and dimensions of the subject being assessed.

An anatomy exam includes questions from all major body systems covered.

Predictive validity

The ability of a test to accurately predict a future outcome or criterion.

SAT scores are evaluated for their ability to predict future college grades.

Concurrent validity

The degree to which a new test agrees with an established measure of the same outcome administered around the same time.

• A new depression assessment agrees with an established depression assessment taken that day.

Convergent validity

The extent to which a measure correlates strongly with other measures of theoretically related constructs.

• Two different anxiety questionnaires have strongly correlated scores. Measures of empathy should go together with other related measures (e.g., sharing)

• Measures of empathy should NOT go together with unrelated constructs (e.g., competitive drive)

Discriminant validity

The extent to which a measure is distinguishable from and does not correlate strongly with measures of theoretically unrelated constructs.

An anxiety test shows little association with a measure of unrelated physical characteristics. People with high empathy scores should have low sociopath scores

Reactivity

A change in participants' behavior caused simply by their awareness that they are being observed or studied.

Employees work harder when researchers watch them.

Demand characteristics

Clues or cues in an experiment that lead participants to infer the researcher's hypotheses or expectations, causing them to alter their behavior accordingly.

Someone figures out that the experiment is about kindness and deliberately acts nicer.

Social desirability

A response bias in which participants provide answers that make them appear favorable or socially acceptable rather than completely honest.

A person lies about how often they exercise to seem healthier.

Nominal scale

A scale of measurement consisting of discrete categories or names with no inherent quantitative order. Example: Eye color, gender, major.

Ordinal scale

A scale of measurement where categories have a meaningful rank order, but differences between ranks are not necessarily equal. Example: 1st, 2nd, 3rd place.

Interval scale

A scale of measurement where intervals between numbers are equal, but there is no true or meaningful zero representing complete absence. Example: Temperature in °F.

Ratio scale

A scale of measurement characterized by equal intervals between values and an absolute, meaningful zero point. Example: Height, weight, age.

Heisenberg uncertainty principle HUP

A fundamental principle in quantum physics stating that pairs of physical properties, such as exact position and exact momentum, cannot both have arbitrarily precise values simultaneously. It is not simply a consequence of a person watching something.

• The more precisely you know where a particle is, the less precisely you can know how fast and in what direction it is moving (its momentum).

Observer effect

The phenomenon where the act of observing or measuring something directly changes or influences what is being observed.

• A researcher observes students during an exam. Because the students know they are being watched, they might act more seriously or avoid checking their phones.

Experimenter effect

An unintended influence on experimental results or their interpretation resulting from the researcher's expectations or behavior.

• What the researcher expects to happen can influence what actually happens or how the results are interpreted.

• A researcher believes one group is smarter and unknowingly provides that group with additional encouragement.

Independent groups design

A between-subjects experimental design in which different participants are assigned to different conditions, so each individual experiences only one level of the independent variable.

• A researcher wants to investigate whether caffeine improves memory.

  • Group 1: Drinks coffee.

  • Group 2: Drinks water.

  • Both groups complete the same memory test.

• Advantages: No order effects because each participant experiences only one condition.

• Disadvantages: Individual differences between participants can affect the results, and more participants may be needed. Can lower this by having randomization

Repeated measures design

A within-subjects experimental design in which the same participants experience and complete every experimental condition.

• Example: Researchers test participants' memory without caffeine and then test the same participants after consuming caffeine.

• Advantages: Requires fewer participants and reduces differences between individuals because each person serves as their own comparison.

• Disadvantages: Order effects can influence performance. Can reduce by using counterbalancing 

Order effects

Changes in participant performance or responses that occur specifically because of the sequence in which experimental conditions are administered.

Practice effect

An order effect where participant performance improves over repeated testing simply due to becoming familiar with the task.

• 
  

Performing better on a memory test the second time because you know how it works.

Fatigue effect

An order effect where participant performance deteriorates over time because the participant becomes tired or bored.

Reaction time gets slower after completing multiple tests.

Carryover effect

An order effect where the influence of a preceding condition persists and affects behavior in a subsequent condition.

• A medication taken during the first condition remains active during the next condition.

Counterbalancing

An experimental technique of systematically varying the presentation order of conditions across participants to distribute and average out order effects.

• Instead of everyone completing A before B, some complete B before A.

• Counterbalancing helps distribute order effects across conditions. It does not necessarily eliminate them, especially if effects persist.

Matched pairs design

An experimental design in which participants are paired based on a shared relevant characteristic and then randomly assigned across different conditions to reduce group differences.

• Example: A researcher wants to study whether tutoring improves exam scores. Two students with similar previous grades are matched. One receives tutoring, while the other does not. This helps reduce differences between experimental groups

Staged manipulation

An experimental manipulation where the researcher creates a realistic event or situation to evoke natural responses from participants.

• Example: Researchers want to examine helping behavior. They arrange for someone to drop a stack of papers and observe whether participants offer assistance. The situation is deliberately staged to create the experimental condition.

Strength of manipulation

The degree of difference introduced by the researcher between conditions of an independent variable to produce a measurable change in the dependent variable.

• Example: A researcher wants to see whether noise affects concentration.

• Weak manipulation: Compare 40 dB with 42 dB.

• Strong manipulation: Compare 40 dB with 85 dB.

Self-report measure

A measurement approach where participants provide direct descriptions or ratings of their own feelings, attitudes, thoughts, or behaviors.

Completing an anxiety questionnaire.

Behavioral measure

A measurement approach based on direct observation and recording of participants' explicit actions, times, or performance.

Measuring how long someone takes to complete a puzzle.

Physiological measure

A measurement approach that records biological or bodily functions, such as heart rate, blood pressure, or neural activity.

Recording heart rate, blood pressure, or brain activity.

Sensitivity

The capability of an instrument or measure to detect meaningful differences or small changes in the variable of interest.

• A scale with only "anxious" or "not anxious" might miss small changes. A more detailed questionnaire with several response options may detect those changes.

• More power

Floor effect

A measurement limitation that occurs when an assessment is so difficult that nearly all participants score near the minimum possible score.

• Floor = Everyone is stuck at the bottom.

Ceiling effect

A measurement limitation that occurs when an assessment is so easy that nearly all participants score near the maximum possible score.

• Ceiling = Everyone is stuck at the top.

Demand characteristics

• Clues in an experiment that allow participants to figure out what the researcher expects, potentially causing them to change their behavior

Placebo

An inactive or sham substance or intervention designed to match the active treatment without containing any therapeutic agent.

Expectancy effect

A change in a participant's state or behavior driven by their belief and expectation regarding what should occur. That expectation may change how alert they feel or perform, even if the pill contains no active drug

Single-blind experiment

A study procedure in which participants do not know whether they are receiving the experimental treatment or the placebo.

Double-blind experiment

A study procedure in which neither the participants nor the administering researchers know group assignments until the study is unmasked.

Complex design

An experimental design that includes two or more independent variables evaluated against a single dependent variable.

Studying how sleep and caffeine affect test scores.

Factorial design

An experimental design that investigates two or more factors (independent variables), evaluating both their individual and interactive effects on a dependent variable.

Testing 2 sleep levels and 3 caffeine doses.

Independent Groups (Between-Subjects)

Different participants are assigned to different conditions. Also called between-subjects design.

• One group drinks coffee, another drinks water.

Mixed factorial design

A factorial design incorporating at least one between-subjects independent variable and at least one within-subjects (repeated measures) independent variable.

Different groups receive coffee or water, but everyone is tested before and after drinking.

Number of conditions (factorial design)

The total number of unique experimental treatment combinations, obtained by multiplying the number of levels of each independent variable (e.g., in a $2 \times 3 \times 2$ design, there are $12$ conditions).

Number of main effects (factorial design)

The total number of possible main effects in a factorial design, which equals the number of independent variables (counted by how many numbers are in the design notation).

Calculating Factorial Designs

How Many Independent Variables?

• Count how many numbers there are

• 2 x 3 x 2 = 3 IV’s

How Many Experimental Variables?

• Multiply the numbers

• 2 x 3 x 2 = 12

How Many Main Effects?

• Count how many numbers

• 2 x 3 x 2 = 3

How Many Conditions?

• Multiply the numbers

• 2 x 3 x 2 = 12

Monotonic relationship

A curvilinear relationship where as one variable increases, the other variable changes consistently in only one direction (always increasing or always decreasing), though the rate of change may vary.

U-shaped relationship

A curvilinear relationship where as one variable increases, the other decreases initially and subsequently increases.

Main effect

The overall direct effect of a single independent variable on the dependent variable, averaged across all levels of the other independent variables.

Simple main effect

The specific effect of one independent variable evaluated at a single, particular level of another independent variable.

Interaction (Moderation)

An experimental outcome occurring when the effect of one independent variable on the dependent variable depends upon the level of another independent variable.

• must have at least two IVs for an interaction

• The provide a more nuanced understanding of effects

• Interactions can reveal a hidden effect