ECON30500 Field Experiments: Unit One

What is a field experiment?

These are experiments that measure the causal impact of a service, policy, program, treatment, or intervention on a measured outcome in a real-life setting, by manipulating treatment to measure changes in the outcome. Ex: Preparing for Life

What is the control group?

The control group is the group of comparison, which does not receive treatment.

What is the treatment group?

The treatment group is the group of interest, as they recieve the treatment.

What is causal inference?

Causal inference is the idea that you are able to find and attribute any difference between a control group and a treatment group, as. they are otherwise randomized, and any variance has to be the result of the treatment.

What are the key steps in conducting an experiment?

1. Design the study.

2. Define eligibility criteria and randomly assign people to either the treatment or control group.

3. Collect baseline data and verify that the assignment looks random.

4. Trial begins, and the program is provided.

5. Monitor the program's implementation to ensure the integrity of the experiment is not compromised.

6. Collect follow-up data for both the treatment and control groups.

7. Estimate program impacts by comparing the mean outcomes of the treatment group with those of the control group.

8. Assess whether program impacts are statistically and practically significant.

9. Test whether the results are valid.

What were the first field experiments?

This was conducted by Newman (1923) and Fisher (1925) who conducted randomized trials in agricultural experiments, so whether the yields were impacted by field conditions.

Who invented the t-test?

This was invented by W.S. Gossett, who conducted agricultural experiments by assigning different conditions to 193 plots of Barley in Ireland, to see the impact on yeastuin in batches of Gness. He invented the Student’ t-test as a way to measure deviations of yeast in batches of Guinness, as he was not allowed to name it after himself.

What was the first wave of field experiments?

This refers to the period between 1960 and 1990, when the US government sponsored large-scale social experiments under Lyndon Johnson’s Great Society initiatives. These examined topics include welfare and training programs, electricity consumption and pricing, housing allowance, bail determination, health insurance, and guaranteed income. There was little academic support, and research was mainly carried out by research organizations.

What is a social experiment?

This is a publicly funded study that is applied over a longer period of time to evaluate the economic and social effects of the experimental treatments. It has four main features:

• Random assignment

• Policy intervention

• Follow up data collection

• Evaluation

What was the second wave of field experiments?

These took place between 1990 and 2010, and saw scaled-back, smaller, and more focused field experiments, which often focused on developing countries, on topics such as poverty reduction, with actors like the World Bank. They were big fans of cluster-level randomization. These were usually led by academics who worked with non-governmental organizations.

What was the third wave of field experiments?

This period started in 2010 up until today, and focuses on the randomization of naturally occurring populations in naturally occurring settings, often without the research participants actually being aware that they are part of an experiment. These are typically carried out opportunistically and on a smaller scale, and designed to test theory rather than provide evidence of policymaking.

What is a natural experiment?

A natural experiment is one where researchers observe the effects of a naturally occurring, independent variable that they cannot control or manipulate.

What was the gift exchange experiment?

Gneezy and List, 2006

This experiment was a natural experiment that was designed to test reciprocity. The first experiment hired 19 workers for a data entry task for $12 an hour. The second experiment hired 23 workers for fundraising work at $10 per hour. After recruitment, treatment groups were told that their pay would be increased to $20 an hour. They found that treated workers worked substantially more; this was short-lived, and the difference became indistinguishable after 2 hours for the first experiment, and after 3 hours for the second. This shows that the gift exchange has an emotional component that disappears over time.

What is an RCT?

This is a randomized control trial, which is a type of scientific experiment that is conducted in the real world to evaluate the effectiveness of a specific intervention.

What are the field experiments of social interventions?

These are often focused on poverty reduction and examine interventions to change how people produce, consume, invest, and save, as well as how they make decisions about their health, education, and reproductive lives.

Example: Mexico PROGRESA

This looked at how welfare benefits to parents were paid, conditional on their children regularly attending school and visiting health clinics. It was evaluated using RCT, and since PROGRESA, similar conditional cash transfers have been implemented and evaluated across dozens of countries.

What are the field experiments on political mobilization?

These include Get-out-the-vote (GOTV) experiments that test the relationships between political communication and voter turnout.

Example: Gerber & Green (2008)

Randomly assigned 30,000 registered voters to receive neutral, mobilization messages via canvassing, phone, or mail in the lead up to the 1998 US election. Researchers used official records to track turnout, bypassing self-reporting bias. They found that face-to-face canvassing increased turnout by 9%, mail increased by 0.5%, and phone calls did not help.

What are the field experiments on prejudice and discrimination?

These included audit and correspondence studies, which aim to uncover discrimination based on social factors such as gender, ethnicity, household arrangements, social class, race, sexual orientation, criminal background, neighborhoods, or prestige of college degree.

Example: Tilcsik (2011)

This study submitted pairs of CVs to 1800 jobs, with half listing the applicant as having been the treasurer of an LGBTQ+ organization and half listing the applicant as having been the treasurer of a neutral political organization. They found that nearly 12% of heterosexual applicants were invited, whereas only 7% of gay applicants were invited.

What are the field experiments on social norms?

These address theoretical questions based on social norms, or shared standards of acceptable behavior within groups.

Example: Keizer et al. (2008)

This study randomly attached flyers to bicycles in the Netherlands in areas with and without graffiti. 69% of people threw the flyers on the ground when the ground was covered in graffiti, whereas only 33% littered in areas without it.

What are the field experiments on incentives?

These experiments explore the role of incentives for achieving desirable behavior. These fall into three categories:

• Prosocial behavior: Using incentives to boost blood donations, survey completion, and charitable giving.

• Educational outcomes: Using incentives to improve educational outcomes.

• Lifestyle habits: Using incentives to promote beneficial lifestyle habits, such as exercising and quitting smoking.

Example: Charness and Gneezy (2009)

This was an RCT that had three groups. Group 1 was given info on the value of sport; group 2 was given info and $25 for going once; and group 3 was given info and $1000 for going 8 times within a month. Group 3 was the most effective; they were most likely to attend the gym and had better health outcomes, which persisted for a few months after the incentives were removed.

What are the advantages of field experiments?

• Can yield compelling evidence of causal effects on real-world behaviors.

• The results of field experiments can advance theory.

• The results of field experiments can inform social policy.

What are the disadvantages of field experiments?

• RCTs can be atheoretical and focus solely on testing whether intervention programs work, rather than why, which limits generalisability.

• RCTs do not necessarily yield evidence superior to that from well-executed observational studies.

• Scalability of an intervention is very difficult as implementation is difficult on a large scale, and the competitive advantage by some participants can shrink when others gain access to it.

What was the energy-saving experiment?

Nolan et al. (2011)

A series of messages written on door hangers was left outside the doors of 1000 participants, and the study examined energy consumption before and a month after. They found that households told their neighbors were trying to save money decreased energy consumption by 8.5%. Money saving, environmental protection, and social responsibility had only a small impact.

How do we measure the causal impact?

The causal impact is measured by comparing the factual (outcome after the program is introduced) and the counterfactual (outcome had the program not been introduced). But we can never observe both conditions, so we must mimic the counterfactual.

What is the counterfactual?

This represents what would have happened to the participants in the absence of the programme.

How do we mimic the counterfactual?

Select a group of individuals that did not participate in the program, which is exactly the same except for their lack of exposure to the program. As such, any differences in outcome are caused by the program.

What was the Balsakhi Program?

Balsakhi Program by Pratham (2002-2003)

Pratham, an NGO, developed a program in which tutors (Balsakhi) helped at-risk children with school work for 2 hours a day, and teachers decided who would get the help.

What are some ways to evaluate the impact of the Balsakhi intervention?

1. Pre-Post: Comparing results before help and after help.

2. Simple Difference: Compare the results of children who got help vs those who didn’t.

3. Difference in Difference: Compare gains in test scores of those who got help and those who did not.

4. Randomized Field Experiment: Randomly assigning all at-risk students to treatment and control groups and comparing these.

Bottom line: which method we use matters!

What is selection bias?

This bias arises from the way we choose who participates. This could be due to eligibility criteria (e.g., using the poverty threshold) or to people choosing to participate or not. This leads to participants and non-participants being different, not just because of the program. It prevents us from reliably estimating the overall treatment effect.

Why is being able to reliably estimate the size of selection bias important?

Without a reliable way, we cannot differentiate between the overall difference into a treatment effect and bias term.

Why does randomizaton solve selection bias?

This makes both groups statistically identical on observable and unobservable variables, so that they do not differ at the start of the experiment.

What is the law of large numbers?

The larger the number of units, the more likely the average of the sample's characteristics will be closer to the expected value, i.e., the more representative of the population it will be. The bigger the sample size, the more similar the treatment and control groups will be.

What is the persistent association correlation?

We always see Y=1 when T=1 and see Y=0 when T=0. But the module focuses on the counterfactual approach.

What is the causal, counterfactual approach?

When we change T from one value to another, then Y changes from one value to another.

What is the potential outcomes model?

This is a model for causal inference.

The first term is the treatment effect we want to find, ie, the effect of treatment on the treated.

The second term is the selection bias, ie, the difference in potential untreated outcomes between the treatment and comparison participants.

What does a positive selection bias look like?

Treatment participants have better results than the control group, even without treatment.

What does a negative selection bias look like?

Treatment participants have worse results than comparison participants even without treatment.

What does a zero selection bias?

Treatment and comparison participants have the same results without any treatment.

How does randomization solve selection bias?

Since treatment is randomly assigned, the only difference between the treatment and control groups is their exposure to treatment. As such, we imply that selection bias = 0.

As such, the average treatment effect is:

What are the three key assumptions underlying randomization?

1. Random assignment of participants to treatment, so that receiving treatment is statistically independent of the participant’s potential outcomes.

2. Non-Interference: A participant’s potential outcomes affect only whether the participant receives the treatment himself/herself, and is not impacted by how treatments are allocated to other patients.

3. Excludability: A participant’s potential outcomes respond only to the treatment, not to other factors correlated with the treatment.

What is a needs analysis?

A needs analysis is an assessment of the problems faced by the target population. They help design the appropriate intervention.

What is a theory of change (ToC)?

It is a description of a sequence of events expected to lead to a particular desired outcome. It consists of:

1. Needs: Problems faced by the target population

2. Inputs: Resources that will be consumed in the implementation of the intervention

3. Outputs: What will be delivered, such as information, a policy, or a service

4. Outcome: Results of interest likely to be achieved once the intervention has been delivered

5. Impact: The chaneg in outcomes that is caused by the intervention being tested

6. Assumptions: Assumptions used to justify the causual chain

What is a process evaluation?

This determines whether the intervention is being delivered as intended.

What is an impact evaluation?

Assess whether causal changes can be attributed to an intervention. Can use experimental or quasi-experimental ways to do this.

What is a cost-effectiveness evaluation?

A way to examine both the costs and outcomes of one or more interventions. It also allows for how much it costs to gain a particular unit of outcome.