Study Notes: Fighting Misinformation on Social Media Using Crowdsourced Judgments of News Source Quality

Core Objective: The study investigates the potential of using crowdsourced trust ratings to fight misinformation on social media by having platform algorithms preferentially display content from news sources that users rate as trustworthy.
The Problem of Misinformation: The rise of social media has created an ecosystem for blatantly false news stories ("fake news") and "hyperpartisan" content (biased or misleading coverage of actual events).
Limitations of Professional Fact-Checking: * Scale and Speed: Creating false content is faster and easier than professional fact-checking. * The "Implied Truth Effect": Failing to tag a false story due to lack of resources may lead users to believe the story has been verified simply because it lacks a warning label. * Focus: Fact-checkers primarily identify blatantly false claims rather than the broader challenge of hyperpartisan bias.
The Proposed Alternative: Wisdom of Crowds: Using laypeople to assess the reliability of news websites (rather than individual stories) allows for a large volume of ratings that can keep pace with misinformation production.

Participants: * Study 1: $n = 1,010$ Americans recruited via Amazon Mechanical Turk (MTurk). * Study 2: $n = 970$ Americans recruited via Lucid, providing a sample nationally representative on age, gender, ethnicity, and geography.
Source Selection (60 domains total per study): * Mainstream Media: 20 outlets (e.g., "cnn.com," "npr.org," "foxnews.com"). Study 2 sources were selected based on high US online traffic. * Hyperpartisan Websites: 22 outlets that produce biased coverage of actual facts (e.g., "breitbart.com," "dailykos.com"). * Fake News Websites: 18 outlets producing blatantly false content (e.g., "now8news.com," "thelastlineofdefense.org").
Measures: * Familiarity: "Do you recognize the following websites?" * Trust: "How much do you trust each of these domains?" (Scale: Not at all, Barely, Somewhat, A lot, Entirely). Ratings were rescaled from the interval $[1,5]$ to $[0,1]$ for analysis. * Cognitive Styles: Participants completed the Cognitive Reflection Test (CRT), which measures the disposition to engage in analytic thought versus intuitive gut responses.
Expert Comparison: Ratings from 8 professional fact-checkers were collected for the sources used in Study 2. These experts showed extremely high interrater reliability (intraclass correlation = $0.97$ ).

Common Ground Across the Spectrum: Despite substantial partisan differences, both Democrats and Republicans rated mainstream sources as significantly more trustworthy than hyperpartisan or fake news sources (P < 0.0001 for all comparisons).
Politically Balanced Ratings: When equally weighting the ratings of Democrats and Republicans to create a "politically balanced" score, every single mainstream source (except "salon.com" in Study 1) was rated as more trustworthy than every hyperpartisan or fake news source.
Partisan Asymmetries: * Democrats trusted mainstream media more than Republicans (Difference: $11.5$ percentage points in Study 1; $14.7$ percentage points in Study 2). * The primary exception was Fox News, which Republicans trusted more than Democrats ( $29.8$ point difference in Study 1; $20.9$ point difference in Study 2). * Republicans expressed higher levels of distrust toward mainstream media in general, possibly due to political discourse or ideological features.
Fact-Checker Alignment: Politically balanced layperson ratings correlated strongly with professional fact-checker ratings ( $r = 0.90$ , P < 0.0001).

Media Source Discernment Score: This was calculated by converting trust ratings for mainstream sources to a z-score and subtracting the z-scored mean trustworthiness ratings of hyperpartisan and fake news sources.
Impact of Analytic Thinking: In both studies, performance on the Cognitive Reflection Test (CRT) was positively associated with the ability to discern source quality: * Study 1: $\beta = 0.16$ , P < 0.0001 * Study 2: $\beta = 0.15$ , P < 0.0001
Ideological Interaction: The positive relationship between CRT and discernment was significantly stronger for Democrats ( $r = 0.24$ ) than for Republicans ( $r = 0.08$ ).
Implications for Reasoning: The data suggests that falling for misinformation/low-quality sources results more from a lack of reasoning than from politically motivated reasoning, as more reflective individuals were better (not worse) at discernment.

Low Familiarity Rates: Participants had low familiarity with hyperpartisan ( $14.5\%$ to $15.5\%$ ) and fake news outlets ( $9.4\%$ to 9.9\%\).
Effectiveness of the Crowd: Excluding ratings from participants unfamiliar with a source actually reduced the effectiveness of the crowdsourcing approach.
Unfamiliarity as a Cue: Participants overwhelmingly distrusted unknown sources. * Study 1: $87.0\%$ of unfamiliar sources were rated below the trust midpoint. * Study 2: $75.9\%$ of unfamiliar sources were rated below the trust midpoint.
Conclusion on Trust: Familiarity appears to be a necessary but not sufficient condition for trust; people are initially skeptical and only gain trust after approving of an outlet's coverage.

Algorithm Design: Platforms could use crowdsourced ratings to down-rank low-quality content. The study suggests using trust ratings in a nonlinear concave fashion: outlets with very low trust should be down-ranked substantially, while differences between high-scoring mainstream outlets should have less impact on ranking.
Alignment with Experts: Concerns about ideological bias in professional fact-checking may be mitigated by the fact that laypeople across the spectrum largely agree with experts about which sites are untrustworthy.
Potential Challenges and Limitations: * New/Niche Outlets: High-quality but less-known sources might be unfairly penalized due to the familiarity-trust link. * Gaming the System: Misinformation purveyors might use domain names that sound credible to deceive laypeople. * Global Application: The studies focused on Americans; cross-cultural research is required for global implementation.