Field Experiments (general theory and examples beyond RCTs)

List, 2011

framed field experiment - resolves issues of lab setting influence

if know being studied and have to opt in = bias in who participates → issue with generalisability

natural field experiments - representative, randomly chosen, non-self-selected

shift from passive observation to deliberate, controlled intervention in natural environments → greater adoption of field experiments

field experiments matter because force researchers to understand everyday phenomena

Bolton and Newell, 2017

while field experiments in policy offer high external validity and societal impact they are often plagued by threats to internal validity and scientific integrity due to institutional environment of government

strengths:

• generalisability

• scale

• societal benefit

weaknesses

• compromised methodology - government designed for service delivery which can force workarounds weakening cause and effect

• replication crisis - difficult due to resources, time and specific populations involved, government may be hesitant to publish null results due to political sensitivity

• pressure of haste and political cycles - bound by political deadlines and staff turnover, pressure for quick fix solutions

• lack of independence - political or economic interest influence

weaknesses of field experiments is not inherent but due to lack of appropriate infrastructure - argue for Goldilocks zone where the intersection of science and policy is just right

• use strategic partnerships, early engagement and standardised procedures

Al-Ubaydli et al., 2017

threats to scalability of field experiments

statistical inference and research practices

• post-study probability - likelihood that a statistically significant finding is actually true

• publication bias - significant results

• proposed solutions - independent replication and journal space for insignificant results

representativeness of the population

• adverse heterogeneity

• selection incentives

• compliance and attrition - experimental settings often have unnaturally high levels of compliance

recommendations

• backward induct - plan research with requirements of large-scale rollout in mind, selecting representative samples, improve science of using science through better stat practices and journal editing

Bouma, 2021

strengths:

• high external validity

• strong internal validity - treatment randomisation

• ex durante evaluation

• assessment of behavioural interventions - impact of nudges, information campaigns or communication strategies

• generate own data

weaknesses:

• scalability

• high cost and time

• limited replicability - difference win demographics

• environmental complexity - evaluating environmental policy difficult due to confounding factors

• lack of mechanistic understanding - show causality but not always clarify the behavioural or institutional mechanism why

recommendation - mixed method approach combining field experiments with choice and lab experiments

Brent et al., 2016

financial rewards (Extrinsic) and moral motivations (intrinsic) are not always additive:

• crowding out - external rewards can decrease prosocial behaviour

• information incentives - in some settings shame is more effective at increasing cooperation

key empirical findings:

• ecological context - physical environment shapes social norms, fishermen working in groups at sea show significantly higher cooperation than those individuals on lakes

• technology + pricing - high dynamic electricity prices only reach peak effectiveness when paired with enabling technology

• social comparisons - most effective for high-usage households

why incentives fail - moral disengagement, signal jamming

challenges for policy - welfare cost of nudges (moral tax), site selection bias by researchers to get biggest change

List and Price, 2016

address valuation bias through cheap talk scripts and consequential designs

WTP sensitive to the mode of elicitation

endowment effect - gap between WTP and WTA tends to disappear as participants gain market experience

social comparison most effective for high-use households

impact of political ideology

real-time feedback complement to pricing - costs more salient

Ferraro, 2009

counterfactual thinking is essential for credible impact evaluation

most current evaluations monitor indicators rather than assessing causal impact

need for attribution - distinguish effects from confounding factors

advocates for experimental designs (randomised trials) to isolate the counterfactual

overcoming unique barriers - focus on intermediate behavioural changes when long-term environmental outcomes are too difficult to measure

even when sophisticated statistical designs are not possible should use counterfactual logic by formulating complex theories of change

Paluk, 2010

randomised field experiments should have a more central role in qualitative research

integrating qualitative methods = identify specific mechanisms of change

• social meaning

• triangulation

• grounded theory - qualitative data can discover new hypotheses and interactions

ways to bridge the gap:

• experimental ethnography

• collaboration between researchers

• sustained research programmes - repeat experiments over time

• create investigator partnerships for high quality field experiments