Methods and tools for policy analysis - Partial

LECTURE 1 - Introduction

Empirical research

Any research that uses quantitative observations from the real world to attempt to answer a research question.

2 main elements of a research project

• Research question

• Research design (one that is adapted to answering the question)

Credibility revolution in social sciences

Emphasis on transparent, credible identification strategies RATHER than purely theoretical models. → Shift toward research designs that approximate experiments.
Correlation → Causal inference

4 questions of a successful research project

• What is the causal relation of interest?

• What is the experiment that could ideally be used to capture the causal effect of interest?

• What is the identification strategy?

• What is your mode of statistical inference?

4 questions of a successful research project: What is the causal relation of interest?

Cause (Treatment) → Effect (Outcome)

• Learning about causal relations helps creating policies

• What would happen under alternative scenratios or policies (counterfactual)

Ex: Does social media increase political polarization? Does education increases wages?

4 questions of a successful research project: What is the experiment that could ideally be used to capture the causal effect of interest?

Name the experiment you cannot run. List them, and it will tell you what your design must imitate, because most of the time, the ideal experiment is not possible.

Ex: Political science: Randomly assign access to social mediat to some individual and not to others.

Ex: Education policy: Randomly assign the amiont of education atteined to individuals.

4 questions of a successful research project: What is the identification strategy?

Identification strategy: How observational data stands in for that experiment.

What are the data, assumptions that make it possible to statistically determine the causal inference.

Ex: Education policy: Use compulsory schooling laws (which is supposed to increaase education) in some countries.

4 questions of a successful research project: What is the mode of statistical inference

4 things to dtermine in advance:

• Population studied

• Sample of this population that will actually be studid

• Statistical assumptions made

• How is the confidence interval constructed?

Counterfactual

The other scenario for the same person. It is impossible, as a matter of fact because you only observe one possibility, you can't compare the same person with itself in another situation.

Ex: Studying at Bocconi vs studying at Catholico university -> Impact on income. So you estimate the other university’s effect.

Ex: Do good political institutions foster economic growth? (Acemoglu, Johnson, Robinson)

1. Research question is “Do good political institutions foster economic growth?”. Initial theory is better institutions lead to more secure property rights and less discretionary policies, which leads to higher investment in human and physical capital → greater eocnomic growth.

2. eza

3. Identifiction strategy: We use the type oif colonization (extractive = low institutions bc want to extractt VS neo-europes = more institutions bc want to build extensions of their countries) to study the difference. Mortality rate of europeans is higher in extractive → So you use this as an INSTRUMNTAL VARIABLE. (Settler mortality shifts institutions without shifting income directly).

4. Mode of statistical inference

• Population: All former colonies

• Sample: 64 former colonies for which all relevant data re available

• Variation: Cross-section

• Regular standard errors

Correlation

Fact about a data. Evolution positively or negatively of one or more variables.

Causality

Causality is the difference a treatment makes compared to what would have happened without it (the counterfactual).

Requires:

• Counterfactual

• Causal model (DGP)

Example: People who took SAT prep course scored 63 points lower than among those who had not.

In fact, prep takers are low ability. They self selected into the group.

But if you compare low ability grop to high ability, you see that prep increases the score but not at the same magnitude.

AVERAGE EFFECT = 0,6 × 400 + 0,4 × 100

Grey numbers are unobserved data (counterfactuals)

Which if these statement is about a causal effect?

Response 1.

Response 2 is about SELECTION into a group.

Response 3 is about CONFOUDING.

Causal model (also called structural model)

Represented by a Directly Acyclic Graph (DAG) containing:

• Variables: Nodes

• Claims: Arrows - Represent a causal effect between two random variables moving in the intuitive direction of the arrow.

The direction of the arrow captres the direction of the causality.

Notation

3 ways 3 variables can be wired

• Chain (Mediator)

• Fork (Confounder)

• Collider (Selection)

Confounding

Treatment and outcomes have common cause, whcih creates a spurious correlation between D and Y.

Backdoor path

Non-causal path between a treatment and an outcome that creates a spurious correlation that muse be blocked to identify the TRUE causal effect.

It "sneaks in through the back" and confounds your estimate.

How to close a backdoor

Conditioning the X = Holding it constant.

How to do that: Subclassification OR matching OR regression OR another method.

How to know if it is open or closed

Open: No collider in the chain.

Closed: A path is closed if it contains a collider YOU DID NOT CONDITION ON.

Conditioning calculation

Treated - untreated within each cells of X, then averages.

Ex of the pre course: using the scores in high school to eliminate past differences to isolate the effect. If there is still a difference, it means it is the causal effect of the treatment.

When is conditioning the good thing to do?

Mediator: It is not mandatory, but possible.

Confounder (fork): Conditioning X is mandatory to close the backdoor.

Colliders: YOU SHOULD NOT condition for X, it would oopen the backdoor.

EXCEPTION: If the X is an unobserved data (not in the dataset) → Then it is a bias that cannot be corrected, backdoor cannot be closed.

What is the consequence of conditioning on a collider

Spurious relationship appears. = Omitted variable bias. → And it is not in our data, we created it by ourselves.

Example of returns to college

ALL THE PATHS TO GO FROM D TO Y:

D -> Y

D <- I -> Y (OPEN PATH)

D <- PE -> I -> Y (OPEN PATH)

D <- B -> PE -> I -> Y (OPEN PATH)

Conditioning on I is the solution to close all backdoors in this situation. I is a non-collider on all three open paths, conditioning on family incomes closes all of them at once, even though B is never observed.

Methods on how to read any DAG

1. List the nodes (including the ones you can’t observe)

2. Draw the arrows you assume

3. State the arrows you exclude

4. Write out eevry path going from D to Y.

5. Mark each path open or closed

6. Close the open backdoors by conditioning.

Backdoor criterion

When all backdoors are closed, the researhc design satisfies the backdoor criterion.

A set of variables X satisfies the backdoor criterion in a DAG if and only if X blocks every path between confounders that contain an arrow from D to Y.

→ Means you have isolated some causal effect.

Another example for backdoor criterion

Path 1 and 2, you want to control for U2 and U1 -> HOWEVER, we can't control for them because they are unobserved, but if they were observed that's what we would have wanted to do

Paths 3 and 4, we don't need to control for anything because I is a collider, so it stops the chain. BUT If we were controlling for I, it would be terrible.

Another example: controlling for discriminations

PAths:

1. F → O → Y (Discrimination via sorting)

2. F → O ← A → Y

YOU SHOULD NOT CONDITION ANY.

1. Conditoining on O would erase part of the effect

2. It is already closed because there is a collider (O), so no need to condition.

3 SLIDO QUESTIONS TO ADD.