Understanding Quantitative Research Designs and Validity

Cross-sectional data collection

Data is collected at a single point in time

Longitudinal data collection

Data is collected at multiple points over time (multiple waves)

Association

Indicates a relationship between variables (X - Y) but doesn't establish that one causes the other

Causation

Indicates that one variable causes a change in another variable (X → Y)

Data points for Association

Usually one data point (cross-sectional)

Data points for Causation

Ideally needs at least two data collection points (longitudinal)

Study design for Association

Cross-sectional, descriptive

Study design for Causation

Longitudinal, causal relationships

Temporal ambiguity

Occurs in cross-sectional studies with association because we can't tell which variable came first - the predictor or the outcome.

Laboratory research

Easier to manipulate IVs and control experimental conditions; can use one data collection point

Field research

More challenging to manipulate IVs and control experimental conditions; typically needs at least two data collection points

Internal validity

The degree of confidence in the study's causal (or cause-effect) relationships

Concern of internal validity

Concerned with the research design and ensuring that causality (X → Y) is real and not caused by extraneous factors

Threats to internal validity

Extraneous variables that may be the true cause of the observed effects on the DV rather than the IV

History

External events that occur during the study that might influence results

Maturation

Natural changes in participants over time

Testing

Effects from taking pre-tests that might influence post-test scores

Instrumentation

Changes in measurement tools or observers

Selection

Differences between groups at the start of the study

Mortality/attrition

Loss of participants during the study

Hawthorne effect

Participants changing behavior because they know they're being studied

Placebo effect

Participants responding to their belief about the treatment

Diffusion of treatment effect

Treatment group influencing control group

Location

Different settings affecting outcomes

Implementation effect

Variations in how the treatment is delivered

Random assignment

Assignment of study participants to groups where each participant has an equal chance of being assigned to either treatment or control group

Importance of random assignment

Distributes bias across conditions; investigator cannot choose the assignment; participants cannot self-select into groups; groups should be similar except for the treatment; allows for blindness (single or double-blind studies)

Experimental study design

IV is manipulated; random assignment; can establish true cause-effect relationships; Example: Randomized Controlled Trial (RCT)

Quasi-experimental study design

IV is manipulated; no random assignment; can establish possible cause-effect relationships

Non-experimental study design

IV is not manipulated; no random assignment; includes all other quantitative research

One-shot case study

(X O) Weaknesses: No pre-test, no control group, many threats to internal validity

One-group pretest-posttest

(O₁ X O₂) Strengths: Can see change over time. Weaknesses: No control group, history and maturation threats.

Static group comparison

(X O / O) Strengths: Has a comparison group. Weaknesses: No random assignment, no pre-test, selection threats.

Posttest-only control group

(R X O / R O) Strengths: Random assignment, control group. Weaknesses: No pre-test to verify group equivalence.

Pretest-posttest control group

(R O₁ X O₂ / R O₁ O₂) Strengths: Random assignment, control group, can see change over time. Weaknesses: Testing effects possible.

Solomon Four-Group

(R O₁ X O₂ / R O₁ O₂ / R X O / R O) Strengths: Controls for testing effects, very rigorous. Weaknesses: Resource intensive, complex analysis.

Interrupted Time-series

(O₁ O₂ O₃ O₄ X O₅ O₆ O₇ O₈) Strengths: Multiple measures, can see trends before and after intervention. Weaknesses: No control group, history threats.

Nonequivalent control group

(O₁ X O₂ / O₁ O₂) Strengths: Has a comparison group, pre-test to verify initial differences. Weaknesses: No random assignment, selection threats.

Survey

Describes attitudes, beliefs, behaviors, etc. using questionnaires. Can be cross-sectional or longitudinal.

Correlational research

Explores associations between 2 continuous variables in one group. No manipulation.

Causal-comparative research

Tries to find causes of existing group differences. No manipulation or random assignment.

Trend study

Different samples from the same population over time.

Cohort study

Same cohort but possibly different individuals over time.

Panel study

Same individuals tracked over time.

Positive correlation

Both variables increase.

Negative correlation

One variable increases, the other decreases.

None correlation

No pattern or relationship.

Curvilinear correlation

Relationship changes direction (e.g., increases then decreases).

Correlation coefficient (r) closer to ±1

Strong correlation.

Correlation coefficient (r) closer to 0

Weak or no correlation.

Causal-comparative research also known as

Ex post facto research.

Causal-comparative non-experimental nature

Non-experimental because there's no manipulation or random assignment; it observes existing differences.

Establishing causality in causal-comparative research

Guidelines like temporal relationship, strength of association, etc. Temporal relationship is the minimum needed (exposure precedes outcome).

Difference between formative/process and summative/outcome evaluation

Formative/process: Evaluates the program while it's ongoing to make adjustments. Summative/outcome: Evaluates final effects after completion.

Inclusion criteria

Characteristics that participants must have.

Exclusion criteria

Characteristics that disqualify participants.

Sampling unit

The individual unit chosen during sampling (e.g., person, group).

Sampling frame

List of all elements in the population.

Theoretical population

All people to whom results might apply.

Study population

Accessible population from which sample is drawn.

Sample

Subset of the study population selected for the study.

Observation

A single measurement or data point collected from one sample unit.

Research flow components

Study population → Sampling → Sample → Generalizability.

Sampling (or standard) error

What are 2 causes and 2 solutions?

Sampling error

Difference between sample statistic and population parameter.

Causes of sampling error

Non-random sampling, small sample size.

Solutions to sampling error

Use probability sampling, increase sample size.

Nonprobability sampling

Not all individuals have a known or equal chance. Used in exploratory or qualitative research.

Probability sampling

Each individual has a known, non-zero chance. Supports generalization.

Simple random sampling

Random selection from entire population (e.g., random number generator).

Stratified random sampling

Population divided into strata (e.g., class year), random sample from each.

Systematic sampling

Every kth person is selected after a random start (e.g., every 10th person).

Cluster sampling

Random groups (e.g., schools or zip codes) are selected, then individuals within.

External validity

Ability to generalize results to other people, settings, or times.

Threats to external validity

Selection-treatment interaction, Setting-treatment interaction, History-treatment interaction.

Selection-treatment interaction

Sample differs from population.

Setting-treatment interaction

Unique setting limits generalizability.

History-treatment interaction

Time or events may affect outcomes differently elsewhere.

Importance of sample size

Helps ensure your sample represents the population, reduces sampling error, increases probability of detecting significant relationships.

Power analysis

A technique used to determine how large the sample must be to detect statistically significant relationships.

Steps of hypothesis testing

Set up hypotheses: H₀ (null) and H₁ (alternative), Collect data and conduct statistical tests, Determine if results are significant, Make decision about null hypothesis.

Effect size

The magnitude of the experimental effect or the strength of a relationship between two variables in a sample.

Measurement

The process of obtaining a numerical description of the extent of a characteristic.

Measurement instrument

A tool used to collect data.

Instrumentation

The process of using measurement instruments in research.

Measurement

Assigning numbers or labels according to a set of rules to measure variables

Measurement instrument

A tool or data collection instrument (questionnaire is most common)

Instrumentation

All measurement instruments used in a study

Quantitative variables

Use standardized data collection (e.g., surveys) with predetermined questions and response options with numeric values

Qualitative variables

Data are language of participants, typically words and narratives, using labels or categories

Self-report measures

A method of collecting data where participants provide information about themselves

Observation

A method of collecting data by watching and recording behaviors or events

Knowledge tests

A method of collecting data that assesses what participants know

Checklists

A method of collecting data using a list of items to be checked off as present or completed

Laboratory tests

A method of collecting data through controlled testing in a lab setting

Strengths of self-report measures

Standardized measures that are easy to administer to many participants

Weaknesses of self-report measures

Bias

Social Desirability Bias

A bias in self-report measures where participants respond in a manner they believe is socially acceptable

Recall Bias

A bias in self-report measures where participants may not accurately remember past events or experiences

Ecological momentary assessment (EMA)

A self-administered data collection method, though no detailed definition is provided