The Architecture of Data Categorization, Codebooks, and Human Bias

Case Study: The Semantics of Emojis in Legal Contexts (September 2022)

• Context: Huntingdon, UK, September 2022. A heated political exchange occurred on X (formerly Twitter) regarding the killing of Chris Kaba.

• Participants:

  • Complainant: Ben Obese-Jecty, a Conservative MP candidate.

  • Defendant: A 26-year-old Black man who replied to Obese-Jecty with two emojis.

• The Prosecution Argument:

  • The emojis were interpreted as a coded racial slur ("c**n").

  • The incident was reported as a hate crime, leading to police prosecution, a house raid, and approximately $10$ hours of custody for the defendant.

  • The central question was whether the reply constituted a joke, a threat, or a coded slur.

• The Defense Argument:

  • The defendant argued the emoji use was a typo intended to be a "clown" emoji, signifying playful mockery rather than hate speech.

  • The defense framed the act as free speech.

• The Verdict (March 2024):

  • The defendant was acquitted.

  • The court accepted the defense's reading of the emoji semantics.

• Implication for Data Thinking: This case illustrates that data points (like emojis) do not have inherent meaning; their meaning is assigned through categorization and interpretation.

The Foundations of Codebooks

• Definition: A codebook is a set of categorization decisions documented in writing so that a stranger can apply them consistently.

• Functions of a Codebook:

  • Translates abstract concepts into measurable variables.

  • Ensures coding decisions are reproducible across different people and over time.

  • Represents a construction that reflects the perspectives and biases of those who built it.

  • Core Philosophy: The purpose of a codebook is reproducibility, not necessarily the "truth."

• The Four Elements of Every Codebook:

  1. Variable: What is being measured or coded (e.g., Age, Sex, Income, Party affiliation).

  2. Codes: The actual values the variable can take (e.g., Male; Female; Other; Prefer not to say).

  3. Decision Rules: Guidelines on how to decide which code to assign (e.g., following self-selection or agreed-upon signals).

  4. Signals: Worked examples and case studies that demonstrate the rule in action (e.g., specific instances used to train coders).

Categorization as a Social Construction

• Foundational Principle: "What gets counted counts." — Joni Seager, feminist geographer.

• The Nature of Categories: Categories are constructions, not discoveries. Every variable in a codebook represents a decision about what counts, reflecting the values of the creator.

• Categorization Process: Deciding what counts as what before any measurement begins.

• Standard Categorization Schemes: Researchers often use pre-existing schemes from established bodies:

  • Age: Pew Research Center brackets (e.g., $18\text{--}24$, $25\text{--}34$, etc.).

  • Education: ISCED 2011 (UNESCO Institute for Statistics) — $9$ levels ranging from Level $0$ (Early childhood) to Level $8$ (Doctoral).

  • Occupation: ISCO-08 (International Labour Organization) — $10$ major groups ranging from Managers to Armed Forces.

  • Generations: Pew Research Center definitions: Boomer ($1946\text{--}1964$), Gen X ($1965\text{--}1980$), Millennial ($1981\text{--}1996$), Gen Z ($1997\text{--}2012$).

  • Marital Status: Eurostat/National census bureau categories (Single, Married, Divorced, etc.).

Case Study: The Evolution of the US Census ($1790\text{--}2000$)

• Overview: The US Census demonstrates how the same variable ("Race") can have different permitted values over $230$ years, reflecting evolving racial ideologies.

• Timeline of Racial Classification:

  • 1790 (First Census): Results gathered for Free white males ($16$ and older), free white males (under $16$), free white females, other free persons, and slaves.

  • 1870: Added "Chinese" as the first Asian category, driven by immigration politics.

  • 1890: The first census to include a complete count of American Indians on tribal land.

  • 1930 (Eugenics Era): Codified the "One-drop rule." A person of mixed white and Negro blood was returned as "Negro," regardless of the percentage. "Mexican" was added as a race for the first and only time.

  • 1950: Categorized Negro, or Black; American Indian; Chinese; Japanese; Filipino; and Other.

  • 1960: A major structural shift occurred as the census moved from census-taker determination to self-selection.

  • 2000: Introduced the multi-race option ($2$ or more races) and separated Pacific Islander into its own category.

• Obsolete Terminology in the Codebook:

  • Mulatto: Generally someone who is black and at least one other race.

  • Quadroon: Someone with "one-fourth black blood."

  • Octoroon: Someone with "one-eighth or any trace of black blood."

• Note on Ethnicity: The U.S. Census Bureau does not consider Hispanic/Latino ethnicity as a race; individuals are asked to select one or more races separately.

Policy Shifts and Political Bias: Meta (January 2025)

• Context: Following November 2024, Meta narrowed the definition of who counted as a "protected" group in its Hateful Conduct policy.

• Comparative Policy Analysis (December 2024 vs. January 2025):

  • Target: Women. Comparing women to household objects was "Hate Speech" in Dec 2024; it became "Permitted" in Jan 2025.

  • Target: LGBTQ People. Calling them "mentally ill" based on identity was "Hate Speech" in Dec 2024; it became "Permitted" in Jan 2025.

  • Target: Immigrants. Comparing them to "filth" or "feces" was "Hate Speech" in Dec 2024; it was downgraded to "Insult" and became "Permitted" in Jan 2025.

• Conclusion: The exact same speech on the same platform received a different verdict based purely on a codebook rewrite.

Architecture Breaks: The Three Gaps in Data Categorization

Gap 1: Humor as an Alibi

• Research: Schmid (2023), "New Media & Society."

• Finding: When hate speech is wrapped in humor, users perceive it as less hostile. Humor cues create "interpretive uncertainty" that allows the content to be treated as benign.

• Mechanism: Codebooks rarely include a variable for "Is this a joke?", allowing humor to serve as a legal or policy-based defense.

Gap 2: Emoji Semantics

• Research: Zhukova & Herring (2024), "Language@Internet."

• Finding: The meaning of emojis shifts across generations, but codebooks are static.

  • Example: Gen Z and non-binary respondents view the same emoji as more passive-aggressive compared to older respondents.

  • Example: The Skeleton/Skull emoji reflects "laughing" for Gen Z (activity slot $4$) while being interpreted literally or negatively by older groups.

Gap 3: In-group Context and Reclaimed Language

• Research: De Kock et al. (IYKYK paper, 2025).

• Finding: The codebook cannot capture relational and contextual signals from outside a community. Even in-group members disagree on the usage of reclaimed language.

• Identity Bias: Annotators (coders) are disproportionately White, Western, and college-educated. Their personal identities shape what they see and define the codebook's blind spots.

Analytical Framework and Midterm Requirements

Five Critical Questions for Data Analysis

1. What variable is being measured?

2. What categories does the codebook allow, and what is missing?

3. Who built the codebook, and when?

4. Who applied it (the coders)?

5. What can't this codebook see?

Midterm Assignment Details

• Task: Select one statistic or dataset and find two sources that present it differently. Analyze what each source counted, framed, or missed using course concepts (DIKW, PPDAC, framing, normalization, codebooks).

• Example (Israeli Tech Sector 2024-2025):

  • Startup Nation Central: Reported over $12$ billion USD in funding.

  • Israel Innovation Authority: Noted record exits but stagnation in output and employment.

  • CTech/Calcalist: Highlighted the "paradox" of more companies but fewer survivors.

  • The Count Variance: Estimates for startups range from $1,231$ to $21,900$ depending on the source's codebook.

• Submission Rules:

  • Length: $3\text{--}5$ pages, double-spaced, 12pt font, 1-inch margins.

  • Due: Wednesday, May 28, 13:00 via Moodle.

  • AI Policy: Permitted but requires disclosure of the model and its use purpose.