ALL Research Design

Small N-Comparative design

hybrid design
combines features within case analysis with logic of large N comparative research

Logic of small N Designs

1) approximate counterfactual
2) selection of cases
3) conditioning via blocking or balancing

How to approximate counterfactual

1) narrow down range of possible relevant factors
2) test specific hypothesis
3) rule out alternative explanations
4) explore internal validity of conclusions
5) question external validity, generalizability

Conditioning via blocking or balancing

keeping everything the same
let MEV vary across cases

Goals of Small N Designs

1) theory testing = retrospectively account for outcomes
2) theory generation = derive hypotheses

Different logics of goals of small n designs

1) deductive logic
2) Inductive logic

Small N cases - Deductive logic

Theoretically motivated research question
researchers select from available cases
confirming/disconfirming the hypotheses

Small N cases - Deductive logic - RQ

does variable X account for outcome Y?

Small N cases - Deductive logic - type of design

MSSD with variation only of main explanatory variable

Small N cases - Inductive Logic

Set of cases
Inference made about the possible causal relationship
new theoretical hypotheses

Small N cases - Inductive Logic - RQ

what explains outcome Y?

Small N cases - Inductive Logic - type of design

MSSD with searching X
MDSD

Most Similar Systems Design 1

focus on one major hypothesised causal relationship with variation on the main explanatory variable
while other values remain constant

Most Similar Systems Design 1 - logic

blocking and conditioning strategy

Most Similar Systems Design 1 - Steps

1) cases are matched on relevant characteristics
2) cases differ in the value of the MEV
3) no matching on mediators or effects of outcome

Most Similar Systems Design 1 - Outcome variable

we do not know the outcome variable before

Most Similar Systems Design 1 - relevant characteristics of cases

confounders
alternative causal factors

Most Similar Systems Design 1 - variables

should be as similar as possible ON alternative causal factors

Most Similar Systems Design 1 - results + interpretation

1) outcome is the same = MEV is not the cause
2) outcome is different = hypothesis that MEV is not the cause not rejected

Most Similar Systems Design 1 - Limitations

1) small measurement errors/random variability = wrong conclusions
2) design cannot accommodate complex relationships

Equifinality

outcomes can result from different processes

conjunctional causation

causal factors are not relevant in isolation but are in certain combinations

Most Similar Systems Design 1 - Address Limitations

1) add more cases
2) add within evidence to evaluate hypotheses

Most Similar Systems Design 2

focus on both values of control and the outcome variable

Most Similar Systems Design 2 - case selection

picking cases that are as similar as possible but differ in the outcome of interest

Most Similar Systems Design 2 - starting points

1) we know that outcome variable differs
2) cases are similar on relevant characteristics
3) we dont know the MEV

Most Similar Systems Design 2 - logic

nothing that is common between the cases can be responsible for the difference in the outcomes
= we have to find smth that is NOT common

Most Similar Systems Design 2 - Steps

1) find very different cases that share the same outcome
2) find what it is that is common between these cases that can account for their shared outcome

Qualitative Comparative Analysis

formalises comparisons among medium number of cases

Qualitative Comparative Analysis - mathematical foundation

Boolean minimisation

Qualitative Comparative Analysis - logic

generalises the idea of the small N comparisons based on varied comparisons

Qualitative Comparative Analysis - theory

set theory

Qualitative Comparative Analysis - Varieties

1) crisp-set
2) multi-variate
3) fuzzy-set

Concepts of causality in QCA

1) conjunctoral/combinatorial causality
2) equifinality

Necessary conditions

if the condition is absent, the outcome will not occur

Sufficient conditions

if the condition is present, the outcome will occur

INUS condition

Insufficient but necessary part of an unnecessary but sufficient condition

Measurement

evaluation of cases with respect to variables

Case

spatially, temporally bounded object, phenomenon or event in the world

A case must be

1) bounded and separable
2) homogenous & stable

Variable

1) operationalised dimension of a concept
2) attribute of a case

Measurement requires consideration of

1) cases (observations)
2) variables of those cases

Levels of Measurement

1) Binary
2) Nominal/categorical
3) Ordinal
4) Interval
5) Ratio

Advantages of Large N Design

1) identify and estimate weak and heterogenous relationships
2) power of many observations to detect a systematic “signal” from the “noisy” data

Caution of Large N Design

1) statistical analysis alone insufficient for causal inference
2) must be coupled with appropriate research design

Pitfalls from Causation to Association

1) causal effect is heterogenous
2) Confounder conceals the association between two variables

Large N design can rule out

1) reversed causality
2) confounders
3) collider bias
WHEN appropriate research design

How does large-N research work?

comparative approach in which evidence across cases (cross-case) is used to evaluate a causal hypothesis

Association between two variables (5 reasons)

1) chance (randomness)
2) X causes Y
3) Y causes X (reverse causality)
4) Z causes X and Y (confounder bias/omitted variable)
5) we condition on a shared effect of X and Y (collider bias)

Causal graphs/diagramms

representations of causal models

DAG - Confounders

DAG - Colliders

DAG - Mediators

DAG - Instruments

Experiment Def

type of research where the researcher has some form of control over the environment

True experiment

randomized controlled trials
researchers control who gets the treatment and who doesn’t and assign the treatment randomly

Quasi-experimental research

there is some form of intervention in the environment but no control of the treatment assignment

Observational research

no control of the environment, only passive observation

Natural experiment

researchers have no control over the environment but there is still random allocation of units into treatment and control
(lottery)

Types of empirical research

Experiments are powerful for

1) Identifying and estimating causal effects
2) Theory and hypothesis testing
3) Studying individual-level mechanisms of aggregate level effects

Steps in experimental research

1) Sample
2) Pre-test
3) assign treatment
4) Deliver treatment
5) conduct post-test
6) Analyse data
7) draw conclusion

Randomness in experimental design

1) selection of sample
2) assignment of treatment

Random assignment as solution to

concerns about internal or causal validity

Random selection as solution to

concerns about external validity (generalisability, transportability) of experimental results

Bigger Sample size relevant for

1) smaller effects
2) more variable effects
3) effects measured with more error
4) effects of treatments with more levels
5) effects of more treatments
6) effects subject to spillover, non-compliance and other compliance

Types of errors

What happens if our study is underpowered?

more chance of Type 2 error

Type 2 error

we conclude that there is no effect, while in reality there is one but our study cannot detect it

Blocked randomization

1) separate participants in groups by some possible confounder (gender)
2) random assignment within each block

Pairwise randomization

1) create pairs of participants that are matched on relevant characteristics
2) randomise treatment with the pair

Types of experimental setting

1) laboratory experiments
2) Field experiments
3) survey experiments

Complications of experimental design

1) noncompliance
2) nonresponse
3) attrition
4) spillover
5) Demand effects

Noncompliance

participants do not follow the course of actin prescribed by their assigned experimental status

Types of participants

1) always takers
2) complier
3) Always-defier
4) Never-taker

Non response

problem of missing data for the outcome variable (post-test) for some participants

Attrition

experiments have several waves and participants quit before the end

Solution to Non response/attrition

model and adjust for it with help of covariates measured before the treatment is applied

Spillover

treatment diffusing from treated units to others

Solution to spillover

define unit of analysis at more aggregate level
collect data to model spillover effects

Demand effects

1) learning
2) social desirability
3) Anticipation

Technique to reduce social desirability bias

List experiments

What kind of causal effects are estimated by experiments?

average treatment effects across treatment and control groups
(difference in post-test scores of two groups)

Alternative estimands

1) intent to treat
2) Effect of treatment on the treated
3) Local average treatment effect

Intent to treat

ATE in a sample with non compliance

Effect on treatment on the treated

ATE among compliers and always takers

Local average treatment effect

ATE among compliers

Limits to experimental research

1) expensive
2) identify only average causal effects, no individual level causal effects
3) not relevant for retrospective questions
4) not suited to study organization and states

Addressing Non-Compliance in Surveys

adjusting intent-to-treat in order to estimate the average effect of intention to treat
relevance depends on purpose (science or policy)

Large-N research counterfactual

is approximated by cross-case data (what has happened to some of the units)
AND assumptions (research design!)

Strategies for causal inference with large-N data

1) Natural experiments
2) Conditioning strategies
3) Mediating
4) Instrumental variables

Conditioning strategies

1) block (keep constant)
2) measure and include in analysis

Mediating

focus on the mechanism

Instrumental variables

find exogenous variation in the explanatory variable (the treatment) that is unrelated to the outcome

Natural experiments - speciality

Random assignment is produced by nature, and not by the researcher

Natural experiments - similar approach

Regression discontinuity

Regression discontinuity

quasi-random threshold that sorts cases in two groups, and examine the variable of interest in these two groups

Natural experiments - Disadvantage

mechanisms are rare

Instrumental variable - steps

1) Chose instrumental variable
2) establish effect of instrument on explanatory
3) effect of explanatory on outcome (residuals from stage 2)

Choose intrsumental variable such that

1) IV is related to X
2) IV is NOT related to Y
3) no common cause Z of both IV and Y