COMM4010 Knowledge Check 2

Restricting minors from accessing tools is a commonly proposed “solution” for addressing harms to youth, but it has downsides. Name three reasons why age-gating is not an ideal approach for addressing harms to minors.

Age-gating isn't ideal because it misattributes harm to technology rather than root societal causes, and verification methods (biometrics, ID, self-declaration) are unreliable. It also creates privacy risks from sensitive data collection, blocks minors from essential online communities like LGBTQ support groups, and leaves them unprepared to navigate harms when they eventually gain access.

What T&S problem does PhotoDNA solve? Provide a basic outline of how it works. [Study recommendation: Hany Farid's materials]

PhotoDNA solves the problem of detecting and removing CSAM and terrorist visual content at scale. It works by converting images into a perceptual hash that remains stable even if the image is resized or lightly edited, then comparing that hash against a database of known illegal content for instant removal.

Define “dangerous speech” and name at least 4 of the 5 elements necessary to evaluate the potential for dangerous speech. [Study recommendation: Susan Benesch's materials; Class #14 slides]

Dangerous speech is any expression that increases the risk its audience will condone or commit violence against another group. It is evaluated across five elements: speaker, audience, social/historical context, medium, and message.

What is a “data void”? (a) Define the term, (b) name 3 of the 5 types, and (c) give at least one example (and explain what type it is). [Study recommendation: “Data Voids” paper]

a) A data void is a search query that returns little to no results, leaving a vacuum that bad actors exploit to push harmful content.

b) Fragmented concepts, outdated terms, and breaking news

c) Ex: "ObamaCare" vs. "Affordable Care Act" is a fragmented concept void, where searches split along political lines despite referring to the same policy.

 In his visit, Yoel Roth argued that hate-speech moderation becomes hardest when users frame prejudice as personal preference rather than explicit hostility.  (a) Provide one of the examples that he used to illustrate this point - and (b) describe why addressing this problem is so challenging.

1. Example: "Swipe left if Indian" (banned) vs. "Only interested in dating Jewish people" (allowed) is the same exclusionary outcome framed differently.

2. Hard because the line between negative exclusion and positive preference is blurry — "Whites only" is technically a positive preference but has hateful historical context. Each case has different circumstances.

In her visit, Camille François argued that ensuring that T&S tools are open-source is essential for the future of the industry. Why? What are the risks in the T&S sector that she thinks open source addresses?

She argues that T&S tools should be open-source because it gives everyone the right to study, modify, and contribute to them

She sees this as essential for addressing GenAI threats specifically, which evolve so rapidly that only shared, community-driven resources can keep pace with building effective safety tools.

In response to the spread of terrorism-related content online, tech companies partnered to launch GIFCT in 2017. (a) What does GIFCT stand for? (b) How does GIFCT know that a piece of content is terrorist and violent extremist content? (c) Describe one potential risk of using this technology to remove terrorist and violent extremist content according to critics of GIFCT. [Study recommendation: Class #16 slides, Gavin Sullivan’s paper]

a) GIFCT stands for Global Internet Forum to Counter Terrorism

b) It’s a tech industry partnership that uses PhotoDNA-style hashing to identify and remove terrorist and violent extremist content by comparing it against a shared database

c) A key criticism is that this automated removal can inadvertently erase evidence of war crimes

What is the business reason that social media platforms prioritize “time spent on platform”? How does this shape the product? Then name one type of tech product that does NOT prioritize this approach and why.

Platforms prioritize time-on-platform because more engagement means more ad revenue and user data for targeting.

This pushes product teams to build addictive features at the expense of user well-being

Google Docs is purely functional and has no ad-based revenue because their product is built for efficiency not engagement.

Why do trust and safety startups compare themselves to Salesforce? In your answer, discuss the role of venture capital and new laws regulating tech companies. [Study recommendation: Lucas Wright paper]

T&S startups want to be like Salesforce (indispensable infrastructure that everyone depends on) because VC investors demand explosive growth that narrow moderation tools can't deliver.

New laws like the DSA, help by creating regulatory uncertainty rather than clear rules, letting vendors step in to define what compliance looks like and sell to a much wider range of customers.

Name 3 ways in which Generative AI alters the practice of Trust & Safety.

1. Increases the speed of content moderation

2. Can reduce human engagement with problematic/scarring content in governance

3. The emergence of deepfakes and impersonation → helps erodes trust in information ecosystem

What is notably different about doing T&S on platforms that involve “real-world” interactions (e.g., AirBnb, Uber) compared to content-oriented platforms? Give an example about what makes this especially hard.

Real-world platforms like Uber or Hinge face T&S challenges that extend beyond the platform itself into physical safety, where harms are hard to verify. For example, if a Hinge match leads to a reported assault, the platform must make enforcement decisions — like banning a user — with no direct evidence of what happened offline.

Camille François approaches propaganda through a framework known as ABC that many of our speakers reference. (a) What does ABC stand for? (b) Apply the ABC framework to the LEGO propaganda videos.

ABC stands for (manipulative) Actors, (deceptive) Behavior, and (harmful) Content. Applied to the LEGO propaganda videos: the actor is Iran-backed Explosive Media, the behavior is disguising propaganda as Western-style AI content, and the harmful content mixes true information with fabrications designed to erode American trust in its government.

What were the two dominant explanations/”sides” for what #GamerGate was? How did #GamerGate alter the field of Trust & Safety?

GamerGate was framed either as a legitimate protest against corruption in gaming journalism or as a coordinated cross-platform harassment campaign targeting women and minorities. It transformed T&S by exposing how harassment could be organized across platforms (4chan, Twitter, Reddit) and how online behavior could escalate into real-world threats like doxxing and swatting.

Renee DiResta argues that “free speech is not free reach.” Explain (a) what this means and (b) how it intersects with the features of social media platforms and content moderation policy.

"Free speech is not free reach" means everyone has the right to express themselves, but not the right to have their speech algorithmically amplified.

This intersects with platform design and content moderation because social media is built around amplification features like algorithms, likes, and shares, meaning platforms can protect free expression while still limiting harmful content's spread through tools like downranking or removing posts from recommendations.

Define moral injury. Why does this framework apply to T&S professionals?

Moral injury is the deep psychological and spiritual distress caused by repeatedly acting in or witnessing ways that violate one's core values. It applies to T&S professionals because constant exposure to traumatic content erodes their sense of meaning, faith in society, and mental health over time.