Notes on Why It's Hard to Know When a Fact Is a Fact

Context: What counts as a fact in social science?

  • Postmodern critique: if you’re a postmodernist, the answer to when something is a fact is often “never.” Facts are shaped by the questions we ask, the viewpoints we hold, and the labels we use. Values (e.g., feminism, civil rights, belief in marriage) influence how we collect data and how we interpret it.
  • Cherlin’s stance: recognizing that questions, categories, and values influence findings does not make all data suspect; it does require critical scrutiny of origins and purposes of facts.
  • Key illustration from family demography:
    • In 1941, Paul Glick categorized American families into 3 groups: (1) normal families, i.e., two-parent families; (2) other male-headed families; (3) other female-headed families. The implication was that single-parent families were “abnormal.”
    • Language matters: terms like “illegal alien” vs. “undocumented immigrant” shape readers’ impressions about people’s legality and worth.
  • Takeaway: Facts should be treated critically; ask about origins, purposes, and relationships to other facts.

Questions to ask when evaluating a fact

  • Who produced this fact? Was it a person or an organization with a particular point of view?
  • What was the purpose of making this fact known?
  • What do we know about the relationship of this fact to other facts or trends?

Case study: familyfacts.org and the Heritage Foundation

  • Familyfacts.org presents itself as a neutral clearinghouse for family research:
    • Home page claim: “The Heritage Foundation's familyfacts.org catalogs social science findings on the family, society and religion gleaned from peer-reviewed journals, books and government surveys. Serving policy makers, journalists, scholars and the general public, familyfacts.org makes social science research easily accessible to the non-specialist.”
  • 2008 feature: a “top ten” list of findings about how children in different kinds of families fare in school.
    • Finding A: “Kindergartners in intact families have higher average reading scores than peers in stepfamilies or cohabiting families.”
    • Finding B: “First-graders whose mothers were married when they were born are less likely to engage in disruptive behavior with peers and teachers than those whose mothers were single or cohabiting.”
  • General pattern: many findings on the site support the view that marriage is best for children and that religion improves family outcomes.
  • Critical observation: these findings are not falsified; they come from reputable studies (e.g., Journal of Marriage and Family) and are described accurately. However, the site selectively reports findings that support a conclusion while omitting those that could complicate or challenge it.
  • Examples of omitted counter-findings from Cherlin’s own work:
    • A 1998 article co-authored with others showed that children whose parents had divorced had a higher risk of emotional problems in adulthood, but not all such problems emerged only after divorce; some were evident in childhood before divorce. (Reference: Cherlin, Chase-Lansdale, and McRae, 1998.)
    • A 1991 article co-authored suggested that some problems for children from divorced families might have occurred even if the parents stayed together. These caveats were not reported on familyfacts.org.
  • Interpretation: Conservative or values-driven groups may promote findings that align with their preferred social arrangements; that does not mean all data are invalid, but it does mean readers must seek broader coverage to get the whole story.

The same dynamic in debates about marriage benefits for same-sex couples

  • The debate around the number of federal benefits and rights tied to marriage centers on the claim of 1{,}049 benefits.
    • In the late 1990s, advocates used the number to argue that marriage confers extensive advantages across policy areas.
    • Hanna Rosin reported in the Washington Post about the claim: “The plaintiffs in the Vermont case documented a long list of benefits granted to married couples but denied to gay ones. The 300 state and 1,049 federal laws cover such matters as the right to pay taxes jointly, and social security benefits."
  • Why the number mattered: The precision of 1{,}049 gave it verisimilitude and made it an attractive symbol for proponents of same-sex marriage; it was widely cited without critical examination.
  • The source of the figure was a GAO study conducted in the context of the Defense of Marriage Act (DOMA). In 1996, Henry Hyde asked GAO to identify all federal laws involving benefits, rights, and privileges dependent on being married. The scope was broadened to include "all those laws in the United States Code in which marital status is a factor, even those that penalize married couples."
  • Concrete examples from the GAO scope:
    • A law limiting certain crop support payments counted married couples as one person; an unmarried couple could potentially receive more because they were treated separately.
    • A campaign-financing rule restricted a candidate’s spending to $50{,}000 of personal funds (or those of close family); but nothing prevented an unmarried partner from contributing additional funds.
  • Practical implication: Many so-called marriage benefits are trivial; yet the number became a powerful symbol in public debates about marriage. The central issue is not whether marriage provides some benefits, but whether portraying the total as a simple, unambiguous list accurately reflects reality.
  • The symbolic power of the number helped shape legal and political arguments about marriage, influencing court decisions and legislative debates.

The role of social status vs. tangible benefits in marriage debates

  • The Massachusetts Supreme Court (Goodrich v. Department of Public Health, 2003) argued that marriage is more than a mere collection of benefits; it is a "status that is specially recognized in society". Denying same-sex couples that status would create a “stigma of exclusion.”
  • In public debates, advocates emphasize the large number of benefits as a way to argue for or against policy changes. The symbol (the number) can be used to simplify complex issues and sway public opinion.
  • The core lesson: debates hinge on how facts are framed and what is highlighted versus what is omitted. Symbols can be more persuasive than nuanced data.

The burden on the careful reader: source, bias, and balance

  • Whether the source is transparent about its positions matters: even well-intentioned friends may have biases.
  • No organization taking sides in a public debate can be perfectly objective; some are more transparent and balanced than others, but all make choices about what questions to ask and what counts as valid evidence.
  • To assess information: examine the source’s stance, look for missing counter-evidence, and seek corroboration from multiple, diverse sources.
  • Bottom line: If you want trustworthy knowledge, you must uncover the source and understand its position on issues. And that careful scrutiny is itself a kind of “fact.”

Practical guidelines for evaluating facts (summary checklist)

  • Identify the producer: Who claims this fact? What organization or person stands behind it?
  • Determine purpose: Is the fact meant to persuade policy, advocate for a position, or inform neutrally?
  • Check scope: What is included or excluded by the definition or measurement? Are there alternative definitions that would change the conclusion?
  • Look for counter-evidence: Are there credible studies or data that challenge the claim? Are they acknowledged?
  • Consider context: How does the fact fit with other data and trends? Is it isolated or part of a broader pattern?
  • Be wary of precision without foundation: A precise number (e.g., 1{,}049) can be appealing, but precision does not guarantee accuracy; verify the source and method.
  • Favor breadth over single findings: Look for a range of studies, methodologies, and contexts rather than a single report.
  • Recognize symbols in debate: Numbers, labels, and categories often serve rhetorical purposes as much as they convey empirical findings.

Notable notes from the text (citations)

  • Glick (1941) — early categorization of families and the implicit normalization of two-parent families.
  • Cherlin, Chase-Lansdale, and McRae (1998) — work cited in discussions of divorce and later emotional problems; shows nuance in timing of problems.
  • Cherlin et al. (1991) — additional nuanced findings about family structure.
  • Snyder in Jencks (1994) — example of advocacy-based estimation processes.
  • Rosin (1999) — coverage of the 1,049 benefits figure in popular media.
  • U.S. General Accounting Office (1997) — the scope and labeling of marital-status-based laws used in the 1999 debate.
  • Goodrich v. Department of Public Health (2003) — legal recognition of marriage as a societal status beyond a list of benefits.

Numerical and factual references to note (LaTeX-formatted)

  • Number of groups in Glick’s typology: 3
  • Time reference for Glick’s article: 1941
  • Year of the 1,049 benefits claim cited in debates: 1999
  • GAO scope development year: 1996
  • Alternative years for related articles and debates: 1998, 1991, 1997, 2003
  • Homelessness estimates discussed: range 2 ext{ to } 3 million
  • The child-reading and behavior findings quoted from 2008 Heritage site: not numerically specified beyond the two examples above
  • The House or court case cited: Massachusetts Supreme Court decision on same-sex marriage rights: Goodrich v. Department of Public Health, 2003

Hypothetical scenario to illustrate the risk of selective facts

  • Suppose a policy brief highlights only studies showing that two-parent families outperform all others in school readiness, while omitting studies showing that children in stable, non-traditional households also fare well in some contexts. Readers would conclude that only two-parent families matter, even if a fuller review shows mixed results depending on support, community resources, and stability. This mirrors how selective reporting can shape public debate despite a broad base of evidence.

Connections to broader themes

  • Theory-ladenness: data are interpreted through theoretical lenses; questions and categories carry assumptions.
  • Epistemic humility: even credible sources may contribute to bias; triangulation with diverse sources is essential.
  • Policy relevance: facts matter for laws, court decisions, and social norms; the way facts are framed can be as influential as the facts themselves.

Quick takeaways

  • Facts are not neutral; their production and presentation reflect values and aims.
  • Always check source, purpose, and the relationship of a fact to other evidence.
  • Be cautious of numbers used as symbols to advance a position; verify the underlying study design and scope.
  • When evaluating social science claims, consult multiple sources and be mindful of what is being omitted as well as what is being highlighted.

References (footnotes from the transcript)

  • 1. Glick (1941).
  • 2. Cherlin, Chase-Lansdale, and McRae (1998).
  • 3. Cherlin et al. (1991).
  • 4. Snyder in Jencks (1994).
  • 5. Rosin (1999).
  • 6. U.S. General Accounting Office (1997).
  • 7. Goodrich v. Department of Public Health (2003).