Democratizing Data. Whose values should guide AI and data practices?

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian

Main argument:
When scientists measure thick concepts, they are inevitably making value judgements about what matters in peoples loves. Because of this. they ought to make these deciosns with the various stakeholders through a structured participatory process that brings brings together lived experience, professional knowledge, and technical expertise on equal footing.

Thick concept measurement is not just a technical activity. It’s a political one.

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian

What is a thick concept? > Elizabeth Anderson definition

A thick concept is a concept that

• is guided by empirical facts

• licenses normative inferences

• its extension is guided by interests and values

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian

3 strategies used to measure thick concepts

1) Turning thick concepts into technical terms

This is the most natural step for scientists trained in the value free ideal of science. They simply strip the evaluative content out of thick concepts and redefine them as purely technical terms.


2) Keeping value judgements in house

Rather than pretending the evaluative content is not there, researchers acknowledge it then take personal/disciplinary responsively for resolving it. They provide normative argument for why their particular operationalisation is the right one.

• Wellbeing example > Psychologists defending life satisfaction as their measure of wellbeing often argue that this is the most democratic option (it allows people to decide for themselves what makes their lives go well, rather than having researchers impose a conception.)

3) Seeking political legitimacy

This approach posits that the judgment should be made through a legitimate political process that includes all those whose lives are affected by the research. > Democratising measurement

• Authors claim this is especially important for when scientific research feeds into public policy and can reshape peoples lives. (eg welfare, healthcare etc)

• This should be through Stakeholder engagement (rather than large scale democratic processes like parliamentary debate etc)

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian
Issues with the strategies in different contexts

The authors do not dismiss strategies one and two.
Strat 1 can be appropriate or highly theoretical work that is far from practical applications

Strat 2 can be is defensible when the precise definition of wellbeing doesn't much matter because an empirical result holds robustly across all plausible definitions.

But when research is close to policy and law, Strats 1 and 2 have serious shortcomings. Strat 1 abrogates researchers responsibility to anticipate misuse of their work. Strat 2 risks imposing researchers’ own value systems on the lives that their research affects.

Hence why in these contexts, Strat.3 is the best.

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian

How measurement of thick concepts works

Characterisation - defining the concept, fixing its boundaries and identifying what features belong to it (ie is it to be assessed on an individual scale or community, what values does it encompass)

Representation - defining a metrical system that captures the quantity or concept (ie an ordinal scale, interval, ratio?)

Procedures - formulating rules for applying the metrical system in practice to produce actual measurement results.


To make this process participatory, the authors claim that stakeholders will need inout at each level

Democratising Measurement: or Why Thick Concepts Call for Coproduction" Alexandrova and Fabian

Co production

The authors introduce their own normative concept > Co production

which is bespoke to the measurement of thick concepts

It refers to the arrangement of sharing power and responsibility in defining thick concepts and developing their measure. It requires:

• Recognising the different types of expertise each stakeholder group brings

• ensuring the final product reflect the demands arising from each type of expertise (as far as possible)

In co production stakeholders have input in the conceptulasiona and the measurement of the thick concept (effectively present at each level)

Wellbeing example > who are stakeholders

• Members of public/service users > expertise insightful in characterisation stage because of their lives experience, meanwhile less insightful in representation as this demands technical quantification.
  

• policy makers. service providers

• scholarly researchers

Overall idea is that all three groups to contribute at all three stages, but with each group's contributions weighted toward the areas where their expertise is strongest

Turn2us case study

Authors use Turn2us as a study of co production

Turn2us is a UK unit poverty charity helping people facing fincnaucl hardship.

The charity wanted to understand what ‘thriving’ meant for its clients and wanted to develop a measure that could track whether their interventions made a difference.

Thriving is a thick concept

Process

Survey 1 → Working Group → Workshop → Survey 2

Survey 1

• distibyed by Turn2us newsletter and received over 1000 responses. It asked respondents about their conception of thriving and what they felt others most commonly misunderstood about thriving for people in their circumstances.

• This survey was designed to give an initial steer to the working group rather than to be the primary vehicle for co-production.

The Working Group

• composed to equally represent all 3 stakeholder groups

• (people with lived experience of financial insecurity, Turn2us staff (professional expertise), and academic researchers (technical expertise)).

• It was small enough to build genuine trust and enable in-depth discussion, but large enough for each form of expertise to be meaningfully represented.

Survey 2

• distributed through Turn2us newsletter > it presented the final theory and was seeking for broader endorsement and testing its representativeness beyond the small working group
  
  
  

LIMITATIONS
The process focused mostly on characterisation — defining what thriving means — but had not yet produced a fully validated scale. Representation and procedures, in the Bradburn et al. sense, remained underdeveloped at the time of writing.

Nonetheless though > shows that co production at least in characterisation stage is a good approach

Turn2us case study issues

• was incomplete > heavily focused on characterisation, not representation or procedures
  
  

• highly context specific > theory of thriving produced was tailored specifically to people experiencing financial hardship in the UK. This depth was a strength in one sense, but it also meant the approach may not scale well to broader national or international measurement efforts where the stakeholder population is far larger and more diverse.
  
  

• Author explans the risk that co-producutin can become a box ticking exerscise providing only the appearance of legitimacy rather than its substance, and can potentially be hijacked by special interests.

Social Science, Policy and Democracy Thoma

Main argument

the argument that democratic alignment is a solution to the value laden nature of social science misses a critical issue. He argues that even if values are democratically selected, a social science built around ONLY those values will leave people who hold different values at an epistemic disadvantage > they will lack the measurements and evidence needed to participate as equals in public deliberation.


The solution Thoma advocates is value pluralism — producing a range of indicators that capture different value perspectives, rather than converging on a single democratically-chosen metric.


In his argument, uses the ONS papers as an example.

• ONS publsuehs CPI (consumer orice index) as the headline inflation measure.

• But anti poverty campaigners argue that it systematically undercounts inflation as experienced by the poorest households, because the standard basket of goods doesn't adequately represent what poor people actually buy (e.g. supermarket own-brand products)

• Also, richer households consumption patterns dominatee the measure

Social Science, Policy and Democracy Thoma

Explaining the nature of thick concepts in social science

It is unontsested that there are thick concepts in social science work. These require normative judgements to be made and thus embed normative commitments.

Thoma notes that it is unrealistic to have complete consensus on these thick concepts (ranging from characterisation [ie what we should measure] to aggregation [how should we weigh different results against each other, eg should each household count equally like it does in CPI]

Social Science, Policy and Democracy Thoma

Technocratic challenge

There is a democratic challenge when social scientists define thick concepts and this is used in public policy > Thoma gives a hypothetical

Social scinticts develop sa national well-being measure entirely on the basis of their own moral institutional without public accountability > Policy makers use this measure to maximise national wellbeing, telling public they are merely following the science

this creates a clear technocratic issue

• power is being exercised by an unelected, unaccountable individual whose value judgments are imposed on everybody.

A popular solution posed to this is democratic alignment. A few measures to achieve this are:

• Accountability > Heather Doiglas argues that scientists should be accountable not just to their scientific communities but to citizens for the value judgments embedded in their work. This is through scientists being chosen by democratically elected decion makers or by transparency in advisory reports
  
  BUT Thoma acknowledges weakeness > both measures of accountability are useless without recourse and dont automatically mean that the research will infirm public deliberation. More robust accountability mechanisms, such as the “science courts” recently proposed by Zeynep Pamuk, may at least in part be able to address some of these concerns
  
  
  

• Well ordered science > Phillip Kitcher argues that hould be oriented toward the public good, understood as what would emerge from ideal democratic deliberation among citizens informed by scientific expertise. The values entering science should reflect those that citizens would endorse under ideal conditions.
  
  BUT > Thoma notes Kitcher has been criticised for the hypothetical nature of his ideal — democratic legitimacy arguably requires ACTUAL participation, not imagined deliberation.
  
  

• Co-prodcution/direct democratic participation > Alexandrova and Fabian argument > Citizens and other stakeholders participate int he deliberation about what values should be captured and scientists defer tot ehoutcome.

Thoma ariuges that all 3 solutions have a common limitation. They treat policy relevant science purely as OUTPUTS that need democratic legitimacy. It focuses too much in ensuring the right values GO INTO science, But it events the queally important lie of scientific outputs as INPUTS into ongoing public deliberation. Science in this lens is used as a resource by cities to hold governments to account and argue their case in public debate

Social Science, Policy and Democracy Thoma

Epistemic inequality

Thoma argues that democratically aligned science can be undemocratic because its creates epistemic inequality

Holiday example
Child is choosing holiday destination with siblings siblings only care about ice cream, child cares to be near water Parents create a dossier. Even if the dossier is democratically decided on (and only includes stuff about ucecrema since that’s what the majority wants) there is still a LEGITIMATE GRIEVANCE. Child doesnt know how well any of the options in the dossier serve his interets thus cannot participate equally in deliberation as the rest of the family.

Thoma’s example maps directly onto social science: the question of what collective priorities should be is separate from the question of what information should be available to everyone when making collective decisions.

Epistemic inequality is a form of epistemic injustice and creates a democratic problem as it creates:

• power imbalances > to know what serves your interets women you can argue more effectively

• violates relational equality

• enables epistemic tyranny of the majority

• entrenches past priorities > once metrics become embedded in institutions, they resist change even as public values shift, because people don't know what alternatives might look like

Epistemic inequality argument (clearly)

When social scientific indicators embed only one value perspective — even a democratically selected one — those who hold different values face a concrete disadvantage:

• The things they care most about may simply not be measured

• They cannot make evidence-based arguments for their preferred policies

• They cannot evaluate how well current policies serve their values

• They cannot hold the government to account on the dimensions they find most important

Thoma > Value pluralism proposal

Thoma argues that social science should aim for value pluralism

• producing multiple indicate that capture different value persoexives so that epistemic goods of scientific research are distbruted more equally.

• This argument is clearly premised on Thoma’s earlier argument that scientific research has importance for its input within public discourse

approaches to pluralism

Dashboard approach

• Rather than aggregating everything to a single number/index, present a range if indicators and leave the weight to users

• New Zealand Living Standards Framework (tracks 12 dimensions of wellbeing without collapsing them into one score.

• Thoma acknowledges that even this dhasbiard approach has some value judgements but it is still more pluralist and abetter approach.

Customisable indicators

• Rather than forcing a single set of weights, let users set their own. The OECD's Better Life Index is the key example: it covers 11 wellbeing dimensions and allows users to adjust weights using interactive sliders, producing a country ranking that reflects their values

Thoma > Democratic benefits to value pluralism

• They make value trade-offs more visible — it becomes clearer that "which policy is better" is partly a question of values, not just facts

• They reduce the scope for politicians to hide behind "following the science"

• They allow citizens to see exactly what compromises a chosen policy involves for them

• They may reveal areas of overlapping consensus — where different value frameworks agree on the same conclusion — making certain decisions easier to justify broadly

• They preserve openness to future challenge — since no single metric becomes so entrenched that alternatives can't be imagined

Thoma > objections to pluaralims approach and rebuttals

Objection 1
complexity will undermine usability and policy guidance
- Thoma argues that the user friendly examples (seen in OCED better life index) show that user friendly pluarims is possible.

Objection 2

Pluralism enables "index-rate shopping" — actors cherry-picking whichever metric suits their interests

ie a business may cite a lower inflation indent when justifying why its not raising workers wages whilst simultaneously citing a higher inflation index to justify raising prices for customers. > choosing whichever figure is convenient. The worry is that pluralism will empower those with power to exploit It and harm the very groups Thoma wishes to protect

Thoma argues that transparency and public accessibility are the solution. If everyone has access to the same pluralist tools, gaming becomes visible and can be called out. Restricting to single metrics to prevent gaming means substituting social scientists' judgment for democratically accountable governments' judgment — which is its own problem. (and arguably a bigger issue)

Objection 3
Pluralism risks partisan epistemology. This is the argument that different political factions will use different metrics with no common ground. The worry is that it will accelerate the fragmentation of shared epistemic ground (something we already see in contemporary democracies)

In a healthy democracy, science is meant to be a neutral tool, commonly accepted by all in the backdrop of public argument. But pluralism can threaten this if different groups start using entirely different metrics. > Partisan epistemology

Thoma argues that the value-ladenness of these indicators is often already obvious, so acknowledging it openly is better than pretending otherwise. > in fact pretending that other values do not exist within a single metric creates more grounds for disagreement and distrust

Crucially, pluralist frameworks still provide a shared framework (the dashboard or customizable tool) that everyone uses — the disagreement is about how to apply it, which is transparently a question of values. This "alignability" — being able to tailor results to your values — could actually build trust in science rather than undermining it.