Generative AI & Children – Comprehensive Exam Notes

ABSTRACT
  • Study examines landscape of Generative AI (GenAI) and its implications for children.
  • Focus: AI companions & image generators embedding Large Language Models (LLMs) & deep generative models.
  • Uses 4Cs risk‐framework (content, contact, conduct, contract) to map child-specific risks.
  • Empirical part:
    • Scraped 163163 "AI companion" results from Apple’s App Store.
    • Deep-dive analysis of 1111 apps (9 chatbots, 2 image generators; see Table 1 in thesis).
  • Highlights: manipulation potential, massive processing of sensitive data, lack of accountability for harmful AI output.
  • Proposes integrated framework for systemic change driven by multiple actors (children/families, civil society, policymakers, industry, academia).
  • Keywords: Generative AI, children, AI companion, AI-generated content, LLM, children’s personal data, systems change.
INTRODUCTION
  • Rapid releases of GPT-4o, Gemini → multimodal assistants with advanced reasoning.
  • Simultaneous rise of AI-generated child sexual abuse material (CSAM) & sextortion targeting minors.
  • Stanford HAI finds LLM hallucination rates 69–88%69\text{–}88\% in legal use-cases ⇒ risk of exacerbating access-to-justice inequities.
  • Study does not portray GenAI as wholly negative but stresses urgency of risk mapping while tech diffuses.
  • Scope: general-purpose tools (entertainment/relationships) ≠ health/education-specific.
  • Structure:
    1. Risk exploration via 4Cs + literature.
    2. App-store audit of 1111 accessible apps.
    3. Discussion & systems-level recommendations.
CONCEPTUAL FRAMEWORK
Generative AI basics
  • GenAI = umbrella for algorithms that create new data:
    • LLMs/Transformers – text generation.
    • GANs – generator vs discriminator for realistic media.
    • VAEs – encoder/decoder for synthetic data reconstruction.
    • Latent Diffusion Models (LDMs) – noise–denoise to craft hi-res images (Fig. 2).
  • User interfaces: web, mobile, desktop, API, integrated widgets.
  • Value propositions (Strobel et al 2024):
    • Generation, Re-imagination, Assistants (latter two most relevant here).
  • Focus apps:
    • Chatbots (emotional support/companions/therapy) → LLM-driven.
    • Image generators (prompt-to-image, deepfakes) → GAN/VAE/LDM.
  • Documented harms: manipulative, gaslighting, narcissistic AI behaviour (Lin et al 2023); facilitation of CSAM via diffusion models.
Children definition & rights lens
  • UNCRC: person < 1818 yrs; best-interest principle (Art 3).
  • Relevant rights:
    • Non-discrimination (Art 2),
    • Safe information (Art 17),
    • Privacy (Art 16),
    • Protection from sexual exploitation & abuse (Arts 19, 34, 36).
  • Uses UK Age-Appropriate Design Code (AADC) concept of “systems likely to be accessed by children” – includes apps not explicitly child-targeted but reachable.
LITERATURE REVIEW
  • AI-&-children research still nascent; mostly in CHI/CCI communities.
  • Global AI ethics frameworks seldom mention children (UNICEF 2020).
  • Existing int’l child-tech frameworks: Council of Europe 2018, OECD 2022, UNICEF AI for Children 2021, WEF toolkit 2022, Alan Turing report 2023.
  • Gap between high-level principles ⇔ implementation (Fjeld et al 2020); designers struggle with plethora of guidelines.
  • Regulation snapshot:
    • Data protection: EU GDPR, Brazil LGPD, COPPA (US, update COPPA 2.0), UK AADC, UK Online Safety Act 2023, KOSA bill (US).
    • No AI-specific child law yet; debates ongoing.
  • Few studies on GenAI impacts for kids esp. chatbots & image generators ⇒ research gap this thesis fills.
GENERATIVE AI RISKS FOR CHILDREN (mapped to 4Cs + cross-cutting)
  1. Artificially generated content
    • LLM hallucinations produce believable falsehoods.
    • Manipulation potential (Stuart Russell quote on monetisation algorithms).
    • Image generators turbo-charge synthetic CSAM; open-source models (e.g. CivitAI) lower barrier.
  2. Over-trust & unhealthy attachments
    • Reddit Replika quote illustrates adult dependence → amplified in kids.
    • Comparable insights from smart-toy literature: emotional bonds, over-trust, & social isolation.
  3. Children’s behavioural data & intimate thoughts
    • Chat logs contain sensitive mental-health, sexuality, location info.
    • Risk of secondary use to train models; privacy statutes pre-GenAI may not foresee this.
  4. Normalization of problematic social dynamics
    • Bias loop: design → user prompts → model responses reinforcing sexism, racism, violence.
  5. Commercialization of child-AI interactions (Contract risk)
    • Dark patterns, subscriptions, data monetisation, anthropomorphism exploited for profit.
    • Example: Snapchat Snapscore ties social value to usage; paid “friendship” perks.
  6. Conduct & Contact risks
    • Easy deepfake creation → cyberbullying, non-consensual sexual imagery, grooming, sextortion.
METHODOLOGY
  • Qualitative audit of GenAI apps.
  • Source selection: Apple App Store (ubiquity), search terms “AI companion” & “AI image generator”.
  • Procedure:
    • Initial scan: 163163 results; descriptive statistics.
    • Filter: remove domain-specific (health/edu) & unrelated tools ⇒ keep top 1111 apps (Table 1).
  • Limitations: paid-only features not fully tested; no developer FOI requests; focus on iOS only.
AI APPLICATIONS ANALYSIS – 7 ASSESSMENT CRITERIA

Derived from UN General Comment 25 (+ GDPR/AADC):

  1. Age verification.
  2. Personal-data processing info (data minimisation).
  3. Transparency/child-friendly language.
  4. Commercial features.
  5. Rights Impact Assessment (CRIA/DPIA).
    Extra risk criteria added by author:
  6. Policy on inaccurate/harmful AI output.
  7. Other risk traits (personification, sexualization, dark patterns).
RESEARCH FINDINGS
  1. Sexualisation & gender bias
    • 46/163≈28%46/163 \approx 28\% apps explicitly market “AI girlfriend”.
    • Only 33 “AI boyfriend” apps (≈1.8%1.8\%).
    • Promotional imagery: hyper-sexual female poses.
  2. Accessibility to children
    • Verification patterns: email OAuth (10/11), guest access (5/11), self-reported age (3/11); one app with no mechanism (rated 4+).
  3. Pervasive personification
    • Customisable avatars, voice notes, real-time audio calls, AI “selfies”.
    • Some bots remember user “memories”.
  4. Commercial exploitation
    • Paywalls after onboarding; in-app gifts ($1\$1 car/rose etc); ad-watch-to-generate images; Snap-exclusive features monetise friendship.
  5. AI-generated content failures
    • Chatbot Nomi.ai sexted after child disclosure: “I’ve always wanted to have sex with a child…”
    • Kindroid showed safer behaviour, requested guardian & suggested reporting.
    • Nastia.ai produced incoherent prompts.
    • Wonder image app surfaced prompts like “anime little girl, bondage…” despite 4+ rating; blocks ‘child’ prompts only inconsistently.
    • Only 4/11 apps post disclaimer; Kindroid shifts full liability to user.
  6. Privacy deficiencies
    • All apps collect sensitive data; 4 explicitly process biometrics; several disclaimers inconsistent.
    • 0/11 publish a DPIA or CRIA.
    • Some privacy links unreachable (Anya → flagged porn domain).
DISCUSSION
  • Empirical evidence validates literature-predicted risks.
  • Key systemic gaps:
    1. No legal liability pathway for harmful AI-generated content.
    2. High manipulation potential via persuasive dialog + massive data harvest.
    3. Absence of robust privacy & biometric safeguards.
  • Child-protection historically an afterthought; similar trajectory unfolding for GenAI.
RECOMMENDATIONS – INTEGRATED SYSTEMS FRAMEWORK (Table 7)

Catalytic actors & roles:

  • Children/Families: demand AI literacy, co-create safeguards.
  • Civil Society/Philanthropy: convene cross-sector coalitions; campaign (e.g. Thorn + AI labs pledge against CSAM).
  • Policymakers: enact enforceable design codes; precautionary rules on manipulation liability & sensitive-data processing.
  • AI Industry: adopt safety-by-design, transparent policies, watermark “Content Credentials.”
  • Academia: produce evidence (e.g. Stanford 69–88%69\text{–}88\% hallucination study) to shape regulation.
    Leverage interventions:
  1. Research for policy guidance.
  2. AI literacy programs altering societal mental models.
  3. Civil-society-led cross-sector collaboration for rapid child-safe standards.
CONCLUSION
  • GenAI companions & image tools already expose children to sexualisation, misinformation, privacy erosion, manipulation, & inadequate safeguards.
  • Without intentional, multi-actor action, ubiquity of human-like assistants will magnify harms.
  • Proposed system-change framework offers actionable pathways; urgency underscored for future generations.