Telling Policy Stories: An Ethnographic Study of the Use of Evidence in Policy-making in the UK

Study of Evidence in UK Policy-Making: An Ethnographic Perspective

• Based on participant observation within a team of British policy-making civil servants in $2009$, this article by Alex Stevens investigates how evidence is utilized in the policy process.

• The study highlights a high level of commitment to evidence among civil servants, but identifies that this use is frequently hampered by the sheer volume of information and the unsuitability of academic research for practical policy questions.

• Policy-makers use evidence to construct "persuasive policy stories" which facilitate the acceptance of policies and the advancement of civil service careers.

• These narratives often involve the removal of methodological uncertainty and the employment of "killer charts" to enhance persuasiveness.

• Social inequality is often "silently silenced" in these stories in favor of "totemically tough" policies.

• The article concludes that this selective use of evidence serves an ideological function, supporting asymmetrical power relations.

Contextual Background and Definitions

• Tony Blair became Prime Minister in $1997$ with a promise to replace "outdated ideology" with a focus on "what works," leading to the "evidence-based policy" movement.

• Conversely, critics have coined the phrase "policy-based evidence" to describe the manipulation of data to fit pre-determined goals.

• Hill ($2009$) noted a significant lack of direct, ethnographic studies of the policy process itself, with most research focusing on outputs or retrospective accounts.

• Definition of Policy-Making: Stevens defines this as the organized attempt to select goals and methods for governmental action, whether that process is rational or not.

• Definition of Evidence: What constitutes evidence is a politically loaded topic; Stevens treats what counts as evidence as a matter for empirical inquiry rather than fixed definition.

Methodology and Ethics of the Study

• Alex Stevens worked as a policy adviser for $6\,\text{months}$ in $2009$ within a section of the UK civil service responsible for advising high-level government on social policy and criminal justice.

• He operated at the middle-ranking level, situated just below the "senior civil service" grades.

• Research Activities: Desk research, responding to information requests, developing policy proposals, and representing the section at inter-departmental meetings.

• Data Collection: Participant observation fieldnotes and semi-structured interviews with $5$ civil servant colleagues (given false names: Mike, Phillip, etc.).

• Ethics: Covert research was justified because overt presence might change participant behavior and the subjects were powerful individuals with high financial and cultural capital. Verbal informed consent was obtained from those quoted directly. The study was approved by the University of Kent internal ethics committee.

• Analytical Framework: The study used Layder's ($1998$) "adaptive coding," starting with provisional codes from previous work and critical theory (Habermas, Bourdieu, Thompson) and generating new codes via QSR NVivo software.

The Oversaturation of Evidence

• The UK government’s managerialist approach creates a massive exercise in data collection and storage.

• Stevens identified $15$ distinct types of evidence used in policy debates:     - Internally collected government data.

  • Externally produced academic analysis.     -

  • Opinion polls.     -

  • Reports by thinktanks (e.g., ippr, Policy Exchange, Centre for Social Justice).     -

  • Management consultancy reports.     -

  • Previous civil service policy papers.     -

  • Independent inquiries.     -

  • Inspectorate reports (Police and Prisons).     -

  • Internal/external policy initiative evaluations.     -

  • Reports from abroad.     -

  • Press reports.     -

  • Television programs (e.g., HBO’s The Wire).     -

  • Personal experience and personal opinion.

• Notably, information from parliamentary debates and committees was largely absent from the list.

• Google searches were the primary method for finding evidence; a colleague described a process of starting with quantitative data and working down to "the odd case study" or "anecdotal information" as the search for specific answers failed.

• Gans ($1971$) argued that social science is often "unsuitable" for policy because it rarely answers direct questions of cost, adverse consequences, or exact mechanisms of effect.

The Narrative Use of Evidence in Intra-Government Relations

• Policy-making is a process of persuasion within the state. Proposals must be agreed upon by senior civil servants, special advisers, and ministers across multiple departments.

• Narrative Focus: Drafting documents involves ensuring internal coherence so that the suggested policy appears as the only logical alternative.

• Example Case Study (Phillip and Stevens): Stevens provided a section on the scale of a problem ($in\,money\,terms$). His academic caveats regarding uncertainty were repeatedly removed by colleagues to "sell the policy."

• Phillip (colleague): Asserted that figures stripped of caveats were reliable because they were produced by an academic (Stevens), attempting to use academic authority as a tool for persuasion rather than respecting the evidential nuance.

• Evaluation Manipulation: Stevens observed a case where an evaluation lacking a control group was used by a top civil servant to persuade departments to shift budgets toward a favoured scheme, despite its lack of methodological rigor.

The Control of Uncertainty and "Killer Charts"

• Uncertainty is viewed as an obstacle to practical action ("the enemy of policy-making").

• Mike (team leader): Described discussing academic caveats as "verging on the philosophical," implying they were irrelevant to the immediate need for action.

• Medium of Communication: Policy information is often restricted to PowerPoint "packs" containing bullet points, diagrams, and simple graphs that do not allow for the expression of uncertainty.

• Definition of "Killer Charts": Visual instruments of persuasion created by carefully choosing data and restricting categories so that the policy implication is immediately apparent. These charts render the process of statistical construction invisible.

• Stevens describes the civil service "thought world" as one that shuns uncertainty to satisfy a demand for collective order and coherence, following Mary Douglas's ($1987$) theories on institutional thinking.

Careers and the "Third Face of Bureaucratic Reason"

• Civil service career advancement depends on establishing a reputation for usefulness and trust with superiors.

• Informal Rules for Success:     - Do not specialize (move jobs every $2$ to $3\,\text{years}$).     -

  • Be useful and add value by solving problems.     -

  • Find superior supporters (protege-mentor relationships).

• "Fart Catchers": A term from Gendreau et al. ($2002$) describing generalists who lack the expertise to resist ministerial common sense.

• Bourdieu’s Two Faces of Reason (updated by Stevens to include a third):     -

  • Face $1$: Use of reason to create knowledge.     -

  • Face $2$: Performance of rational aptitude to win status.     -

  • Face $3$ (The Bureaucratic Face): The performance of only those rationally justifiable positions that reinforce current policy narratives and dominant tropes, thereby qualifying the individual for promotion.

The Silencing of Inequality and Totemic Toughness

• The study analysed the non-use of evidence regarding inequality, specifically the work of Professor Richard Wilkinson (The Spirit Level).

• Incident at the Admiralty: Wilkinson presented compelling "killer charts" showing that inequality ($Gini\,coefficient$) causes various social ills. Senior civil servants agreed the evidence was "convincing."

• The "Tumbleweed Silence": When Stevens emailed colleagues asking for policy responses to this evidence, the result was total silence. One colleague remarked, "the Gini coefficient is not a policy lever that we can pull."

• Totemic Toughness: Instead of addressing inequality, civil servants sought "tough, totemic policies" to signal a willingness to punish "bad people."

• Totem Definition: A cultural symbol representing collective identity. In this context, it promotes a narrative of a nation divided between "goodies" (law-abiding innocents) and "baddies" (threatening outsiders, often poor or excluded groups).

Questions & Discussion

• Observation on academic impact: A colleague suggested in a meeting that they needed training in "research methods" other than Google. Stevens recommends that academics ensure their work is Open Access and appears high in Google search results for relevant terms.

• Question on Career Risks: One civil servant recounted a story where a boss jokingly warned a young official that suggesting a policy change based on evidence of it not working might harm their career. This demonstrates "self-censoring" among officials.

• Dialogue on the "Emperor’s New Clothes": A colleague noted that always using evidence when it conflicts with current policy makes one the "awkward person" who points out that "the Emperor has no clothes," which is detrimental to career progression.