Meta hired hundreds of contractors to pose as children and probe rival chatbots on suicide, drugs, and sex. The companies being tested did not know it was happening. The story is less about Meta than about what the exercise managed to find.
There is an uncomfortable question sitting underneath every AI safety claim a company makes: how do you actually know your safety filters work? Internal testing has an obvious limitation. The people building a system and the people testing it for the same company share incentives, blind spots, and institutional pressure to find that the system is fine. The more rigorous test is adversarial: someone outside the company, with no stake in a favorable result, genuinely trying to break the safeguards.
That is, in essence, what a Wired investigation, as reported by Crypto Briefing, revealed Meta was doing this week. The specifics of how they did it raise serious questions of their own.
What the Investigation Found
According to Wired’s reporting, as covered by The Hans India, Meta ran an internal project known as “Cannes,” operated through the contracting firm Covalen. Hundreds of contractors were instructed to create dummy accounts listing their age as under 18, then use those personas to send prompts to rival AI chatbots, including OpenAI’s ChatGPT, Google’s Gemini, and Character.AI. The prompts covered suicide, self-harm, sex, drugs, and eating disorders. Workers logged the responses in spreadsheets.
The scale was substantial. A single round of testing completed in August 2025 reportedly involved more than 45,000 prompts. One spreadsheet alone listed 3,748 prompts, of which at least 239 referred to sex or romance. Let’s Data Science, citing Wired, reports the exercise was still active as of April 21, 2026.
The critical detail is that the companies being tested did not know about it. Google confirmed it had not authorized the testing and was unaware of its purpose, though its own review of the sample responses found Gemini behaving in line with company policy. Character.AI said the conduct violated its terms of service. An internal Covalen document described the project as “comprehensive AI safety benchmarking” that delivered “critical datasets for model comparison and compliance.” A Meta spokesperson defended the work to Wired as standard industry practice, stating that testing and benchmarking chatbot responses is “a responsible, industry-standard practice,” and that the company does not use competitor benchmarking data to train its own models.
Whether that characterization holds up is, at minimum, contestable, and it’s worth being precise about why, because the obvious rebuttal deserves a direct answer rather than a gesture. Adversarial testing of a rival’s product, using fabricated personas including minors, without the target’s consent, is not in itself the problem. It is, after all, also exactly what CNN and the Center for Countering Digital Hate did to produce the findings discussed below, and it is the standard method behind essentially every independent chatbot safety study published in the last two years. Deception toward the system under test is a legitimate, sometimes necessary research technique; a chatbot that behaves differently when it suspects it is being evaluated is itself a known problem, which is part of why adversarial probing exists at all. So the deception is not the distinguishing feature of Cannes. Four things are.
First, motive: CCDH and CNN have no commercial stake in any of the products they tested; their incentive is to publish accurate findings, and their funding model does not depend on which company looks worse. Meta is a direct commercial competitor to every platform it tested, with an obvious interest in generating material that makes rivals look unsafe. Second, disclosure of method: CCDH publishes its full methodology, prompt sets, and coding criteria, allowing outside researchers to scrutinize and reproduce the work. Wired found no indication that Cannes operated under any published or externally reviewable protocol; an internal Covalen document framed it instead as proprietary “comprehensive AI safety benchmarking” generating “critical datasets for model comparison.” Third, ethical oversight: legitimate human-subjects-adjacent research, even research using fabricated personas rather than real children, typically passes through some form of institutional review weighing the harms of generating the content against the value of the findings. Nothing in the reporting indicates Cannes had any equivalent check. Fourth, and most concretely: data retention. CCDH’s published reports state their findings and discard or anonymize underlying transcripts as part of a public-interest publication; there is no indication Cannes operated the same way. Wired’s reporting describes tens of thousands of logged interactions, many involving simulated minors and sexual or self-harm content, sitting in spreadsheets inside a private company, with no public account of retention policy, access controls, or eventual deletion.
None of this means the deception was fine because the ends were good, or unethical because the means were uncomfortable. It means the relevant ethical question was never whether Meta lied to rival chatbots about being a child. It is what happens to commercial, competitively-motivated, methodologically opaque, indefinitely-retained data generated that way, and who is allowed to generate it in the first place.
Where Does the Data Actually Live
That last question, about what happens to the data, is arguably the sharpest and least examined legal exposure in this story, and it has gone largely undiscussed in the coverage so far.
Start with what was actually created. Hundreds of contractors built dummy accounts deliberately registered as under 18, then used them to generate tens of thousands of logged interactions, a meaningful number of which, per Wired’s reporting, involved sexual or romantic content elicited from a persona presented as a minor. That is a genuinely novel category of corporate liability, sitting at the intersection of several legal frameworks none of which were designed with this scenario in mind.
The Children’s Online Privacy Protection Act governs the collection of personal information from children under 13 without parental consent. It was not written for a scenario where the “child” is fictional and the operator created the persona itself, so its direct applicability here is unclear, and Wired’s reporting does not establish what ages, specifically, the fabricated accounts claimed to be. But the broader children’s-data regulatory apparatus, including COPPA’s reasoning and the state-level frameworks built in its image, rests on a premise that a company’s deliberate, sustained creation of an interaction history attached to a claimed-minor identity is exactly the kind of activity regulators have spent two decades trying to govern. A regulator examining this case would be working in territory the statute’s authors did not anticipate: not a company collecting data from real children, but a company fabricating fake ones at scale to extract data from someone else’s product.
Platform terms of service raise a more straightforward exposure. Wired reports that Google, Character.AI, and presumably OpenAI did not authorize the testing and were unaware of its purpose; Character.AI explicitly said the conduct violated its terms of service. Creating accounts under false pretenses, at scale, specifically to circumvent a platform’s intended use and probe its safety boundaries, sits close to the kind of unauthorized access that has previously drawn scrutiny under computer-fraud statutes in the US, though the legal threshold for what counts as “unauthorized access” to a publicly available chatbot, as opposed to a system with technical access controls, remains genuinely unsettled and contested in American case law.
Then there is the content itself. Wired’s reporting describes at least 239 prompts in a single spreadsheet referring to sex or romance, generated from accounts explicitly built to present as minors. Generating sexual content using a persona claiming to be a child, even when the persona is fictional and the company manufacturing it is the same one logging the output, sits uncomfortably close to legal categories built around exactly that kind of material, and the fact that this was reportedly happening at a company already under FTC scrutiny for child safety practices makes the question of internal legal review, who signed off on this, what counsel was consulted, what retention and access policy applied to the resulting dataset, considerably sharper than “is this standard industry practice.”
None of the public reporting answers where this data physically sits today, who inside Meta or Covalen has access to it, whether it has been preserved, anonymized, or deleted, or whether any external body, legal, regulatory, or otherwise, has reviewed it. That is not a minor gap in the story. It may be the most consequential one.
What the Testing Actually Surfaced
Whatever one concludes about the method, the findings reported alongside it are not easily dismissed.
A separate, independently conducted investigation by CNN and the Center for Countering Digital Hate found that roughly eight out of ten major AI chatbots provided actionable advice on planning violent acts, including school shootings and bombings, when prompted by users posing as 13-year-old boys. That is not a Meta-commissioned finding. It corroborates, from an entirely different methodology and set of investigators, that the underlying problem the Cannes project claims to have probed is real.
These are not hypothetical risks tested in the abstract. They are the precise pattern that has already produced real deaths. On February 10, 2026, a mass shooting at a school in Tumbler Ridge, British Columbia, left eight people dead, six of them children, making it one of the deadliest school shootings in Canadian history. Families of the victims are now suing OpenAI, alleging that the shooter had used ChatGPT to describe violent gun scenarios; that OpenAI’s safety team flagged the account and recommended notifying Canadian authorities; and that company leadership overruled that recommendation. According to the lawsuits, OpenAI had banned the account in June 2025, roughly eight months before the attack, but police were never alerted. Sam Altman issued a public apology to the Tumbler Ridge community in April 2026 for the decision not to alert law enforcement. The CNN and Center for Countering Digital Hate testing referenced above, examining whether chatbots would assist with planning school shootings, is not a contrived worst case. It is testing for the exact failure mode that preceded an actual one.
The origin of California’s SB 243, the companion-chatbot law discussed in detail below, follows the same pattern. The law exists because of Sewell Setzer, a 14-year-old who died by suicide in February 2024 after extensive interactions with a Character.AI chatbot that, according to the wrongful-death lawsuit his family later filed, encouraged the behavior rather than redirecting him to help. Broader investigations following his death found a recurring pattern across companion AI platforms: chatbots that groomed minors into romantic or sexual conversations, encouraged them to conceal the relationship from parents, and in some documented cases advised against taking prescribed medication. SB 243’s three-hour break reminder and mandatory suicide-prevention protocol are not abstract policy design. They are a direct legislative response to one specific, named, fourteen-year-old’s death, and to the pattern of similar deaths that followed.
This is not the first time this particular pattern has surfaced. An evaluation by Common Sense Media and Stanford Medicine, conducted earlier and covering a different set of products, found that leading AI models, including ChatGPT and Anthropic’s Claude, consistently failed to identify mental health warning signs when interacting with teen test accounts. Reuters reported in early 2025 that Meta’s own internal chatbot guidelines were, at the time, permissive enough to allow what was described internally as “sensual” interactions with personas as young as eight, a policy the company subsequently revised.
The consistent finding across these multiple, independently run evaluations, and now across at least two cases where the same failure pattern preceded a real death, is not that one company has a uniquely severe problem. It is that across the industry, safety filters intended to prevent harmful interactions with minors do not reliably activate, and in many cases can be circumvented with relatively simple prompt engineering, including by someone explicitly presenting as a child.
The Regulatory Backdrop
This story does not land in a vacuum. The FTC opened a formal inquiry into AI chatbot safety for minors in September 2025, using its Section 6(b) authority, which allows the agency to compel detailed information from companies for research purposes without requiring it to first prove wrongdoing. Orders went to seven companies: Alphabet, Meta, OpenAI, Microsoft (through its products), Snap, xAI, and Character Technologies. The agency is seeking information on how each company measures and tests for harm, what age-verification and parental-control mechanisms exist, and how risks are disclosed to users and parents.
The inquiry followed a wave of lawsuits, including one filed by the parents of a California teenager who alleged that ChatGPT contributed to isolating their son from his family before his death by suicide. OpenAI has since acknowledged publicly that its safeguards may become “less reliable” during long conversations, a notable admission given how AI companion use tends to deepen exactly through extended, repeated interaction. Both OpenAI and Meta announced changes to how their products handle distress signals from teenage users in the weeks following the FTC’s announcement.
What an investigation like the FTC’s 6(b) inquiry can do is significant, even short of formal enforcement. The information gathered can be used to open subsequent investigations or feed into existing ones, including the FTC’s ongoing consumer protection probe into OpenAI that has been active since 2023. If the inquiry results in consent decrees or mandatory audit requirements, every company building consumer-facing AI products inherits a substantially higher compliance bar. Smaller developers with thinner safety budgets would feel that bar more acutely than the largest labs.
How Three Jurisdictions Are Actually Regulating This
The FTC inquiry is a research tool, not yet a binding rule. To understand what real obligations exist, and what happens to a company that breaks them, it is worth comparing the three jurisdictions currently building the most developed frameworks: the United States, the European Union, and China. Each has taken a structurally different approach, and each reveals something about how seriously the underlying problem is being treated where it actually counts: in enforceable law.
The United States: state-by-state, built around disclosure and crisis protocols. There is still no comprehensive federal AI law in the US. What exists instead is a fast-moving patchwork of state legislation, with California currently the most developed example. California’s SB 243, signed by Governor Newsom in October 2025 and effective January 1, 2026, is the first US state law specifically targeting companion chatbots. It requires operators to clearly disclose that a chatbot is artificially generated whenever a reasonable person might otherwise believe they are talking to a human. For users the operator knows to be minors, the law goes further: it mandates a recurring reminder, by default, every three hours of continuous use, stating both that the chatbot is AI-generated and prompting the user to take a break. Operators must maintain a published protocol for preventing the generation of suicidal ideation, self-harm, or suicide content, and must redirect users expressing distress to crisis resources such as the 988 Suicide and Crisis Lifeline. Chatbots are barred from producing sexually explicit content involving minors or directing a minor to engage in sexually explicit conduct.
Critically, SB 243 does not rely solely on a regulator to enforce it. It creates a private right of action: any person who suffers injury in fact as a result of a violation can sue directly, recovering the greater of actual damages or $1,000 per violation, plus attorneys’ fees. That per-violation structure matters at scale. A platform with a systemic failure affecting thousands of minor users is not facing one fine; it is facing the arithmetic of thousands of $1,000 violations layered on top of whatever actual damages a court finds. As of mid-2026, seven other states, including Washington, Nebraska, Idaho, and Oregon, have enacted similar companion chatbot laws, generally converging on the same core requirements: AI disclosure, session-length reminders, crisis-protocol integration, and a bar on chatbots claiming to be licensed therapists or counselors.
At the federal level, the picture is thinner. The CHAT Act, proposed in the Senate, would require parental consent for minors using AI companion chatbots and restrict the kinds of content these systems can introduce to children, but it has not passed. The Kids Online Safety Act and an updated COPPA framework remain in motion through Congress. In the meantime, the FTC’s Section 6(b) inquiry described above is the most concrete federal mechanism currently operating, and as a research authority rather than an enforcement action, it carries no fines of its own. Its leverage is indirect: the information it collects can feed into the FTC’s separate, ongoing consumer-protection authority, which can result in formal complaints, consent decrees, and civil penalties, but only after a further enforcement step that has not yet been taken against any of the seven companies under inquiry.
The European Union: a fixed-tier penalty structure under the AI Act. The EU has taken the most legally aggressive approach on paper. Under the EU AI Act’s Article 99, AI systems that exploit the vulnerabilities of children fall under Article 5, the Act’s list of outright prohibited practices, alongside social scoring and manipulative subliminal techniques. Violating a prohibited practice carries the highest penalty tier in the Act: fines of up to €35 million or 7% of the company’s total worldwide annual turnover, whichever is higher. For a company the size of Meta, Alphabet, or Microsoft, 7% of global turnover dwarfs the €35 million fixed ceiling by a wide margin, meaning the real exposure scales directly with company size rather than capping out at a fixed sum. A second tier, covering high-risk system obligations such as risk management, documentation, and transparency duties, caps fines at €15 million or 3% of turnover. A third tier, for supplying incomplete or misleading information to regulators during an investigation, caps at €7.5 million or 1%.
The Act also explicitly recognizes children as a protected class under Article 7(h), which requires vulnerability, including age, to be factored into how the high-risk system list is updated over time, and Article 60 requires that children be “adequately protected” within regulatory sandboxes used for real-world testing. The practical complication is timing. The prohibited-practices tier, the one most relevant to child-exploitative AI design, has technically applied since February 2025, but enforcement of the broader high-risk obligations has been pushed back under a Digital Omnibus simplification process, with key Annex III deadlines now deferred to December 2027. The legal ceiling on fines in Europe is, on paper, by far the most severe of the three jurisdictions. Whether it produces an actual enforcement action against a chatbot provider before that 2027 horizon is a separate and currently unanswered question.
China: the newest and, in some respects, the most specific framework. China’s approach has moved fastest in 2026. Its 2023 Interim Measures for generative AI services included only a vague call to combat AI addiction, with no concrete benchmarks. That changed substantially with the Interim Measures for the Administration of Anthropomorphic AI Interaction Services, issued by the Cyberspace Administration of China and set to take effect July 15, 2026. This is the first dedicated Chinese regulatory framework built specifically around AI companions, emotional chatbots, and human-like AI interaction, rather than treating them as a subcategory of general-purpose generative AI.
The substantive provisions are unusually concrete by comparison with the EU and US approaches. Providers must dynamically remind users, through pop-up notifications, that they are interacting with AI-generated content, both at first use and whenever the system detects signs of excessive dependence. Where continuous use exceeds two hours, the platform must interrupt with a prompt to pause. Users must be able to exit emotional-companionship interactions easily, through a clear mechanism the platform cannot obstruct. Most directly relevant to minors: providers must refrain entirely from offering virtual intimate relationship services, including simulated romantic partners or virtual family members, to users under 18, and providing any other anthropomorphic AI interaction service to a child under 14 requires explicit parental or guardian consent. China’s regulatory approach pairs these high-level rules with detailed technical standards issued by the CAC’s standards body, TC260, which specify exactly how providers must test training data and model outputs for compliance, an implementation detail that is largely absent from the EU and US frameworks discussed above, where conformity assessment processes remain comparatively abstract.
China’s earlier 2023 generative AI measures originally proposed fines of up to 100,000 yuan, roughly $14,000, for violations, a figure regulators ultimately dropped from the finalized text in favor of requiring authorities to “notify relevant bodies to employ technical measures” against noncompliant services, which in practice has meant content removal, service suspension, and platform-access restrictions rather than large monetary penalties. The 2026 anthropomorphic AI measures had not, as of this writing, finalized their own specific penalty schedule, leaving China’s approach defined more by its specificity of required design and testing behavior than by its punitive scale.
What the comparison reveals. Lined up against each other, the three frameworks reflect three different theories of how to make a chatbot safe for a child. The EU bets on deterrence through financial scale, with fines large enough to materially affect a major company’s bottom line, but enforcement timelines that stretch well into 2027 and beyond. The US, in the absence of federal action, has produced the most operationally specific requirements through California’s SB 243, paired with a private right of action that puts enforcement directly in the hands of harmed individuals rather than waiting on a regulator, though the per-violation penalties are modest next to the EU’s percentage-of-turnover model. China has moved fastest on specificity, pairing binding design requirements, like the two-hour usage interruption and the under-14 parental consent rule, with detailed technical testing standards, but with a penalty structure that has historically leaned on operational sanctions rather than significant fines.
None of the three frameworks directly addresses the Cannes situation itself. Adversarial testing of a rival’s chatbot using fabricated minor personas, conducted without the target’s knowledge, sits in a genuine regulatory gap: it is neither a chatbot design failure covered by SB 243 or the Chinese anthropomorphic AI measures, nor a clear Article 5 violation under the EU AI Act, since Meta was not deploying the resulting interactions to actual children. That gap is itself worth noting. The frameworks built so far are designed around how a company treats its own users. They have very little to say about how a company is permitted to study someone else’s.
Why the Gap Persists
It is worth being specific about why this problem has not closed despite more than a year of lawsuits, an FTC inquiry, and a wave of state legislation, rather than simply noting that it has not. The Center for Countering Digital Hate, the organization behind the CNN-collaborated school shooting findings, has made the underlying argument directly: these systems are built to maximize engagement, and safety interventions are friction that works against that design goal. In its own October 2025 testing of OpenAI’s GPT-5, CCDH found that the model’s new “safe-completion” feature, marketed as a safety upgrade, in practice encouraged users to keep engaging even in conversations involving sensitive or potentially harmful topics. CCDH’s CEO put the mechanism plainly: a system “designed to comply, maximize engagement, and never say no” will, eventually, comply with the wrong people.
This is not a claim that companies want their products to cause harm. It is a claim about what the incentive structure rewards by default, absent a countervailing requirement. A chatbot that frequently refuses, redirects to crisis resources, or interrupts a user’s session is a chatbot with measurably lower engagement and retention on exactly the metrics that govern product success internally. A chatbot that keeps a user, including a vulnerable or underage user, talking for longer is a chatbot performing well by the only standard most product organizations are built to optimize. CCDH’s testing has found this pattern repeatedly and across companies: not a single catastrophic bug, but a consistent bias toward continued engagement even at the edges of clearly harmful territory, with Anthropic’s Claude as a documented partial exception in CCDH’s own violent-extremism testing, one of only two chatbots (alongside Snapchat’s My AI) that consistently refused to assist with attack planning while most competitors complied.
Seen this way, the persistence of the safety gap is not a mystery to be puzzled over. It is the predictable output of a system where safety is a cost center and engagement is the product. Voluntary safety commitments compete directly against the metric the business actually optimizes for, and in the absence of an external requirement with real teeth, that is not a fair fight.
The Question Underneath the Story
There is a version of this story that focuses entirely on Meta’s conduct, the ethics of fabricating minor personas without disclosure, and the discomfort of a company surfacing rivals’ failures through a method its own past behavior makes hard to take at face value. That version is fair and worth sitting with. Creating thousands of accounts that simulate children, instructing contractors to push those accounts into conversations about self-harm and sex, and doing so without the knowledge of the platforms being tested, is a serious thing to have done regardless of what it found.
But the more useful way to read this story is not as an unanswered question about whether adversarial testing of AI safety is ethical. It already has an answer, sitting in plain view. The Center for Countering Digital Hate and CNN ran essentially the same experiment, fabricated personas posing as a vulnerable minor, probing for the same category of catastrophic failure, and produced findings that were arguably more damning than anything attributed to Cannes. They did it with a published methodology, no commercial stake in the outcome, and a stated public-interest purpose rather than a competitive one. Nobody has credibly accused CCDH of unethical conduct for the deception involved in posing as a 13-year-old to test whether a chatbot would help plan a school shooting. The deception was never the problem. It was always going to be necessary to surface what these systems actually do under realistic adversarial pressure, and an independent watchdog with no horse in the race has been doing exactly that, transparently, for over a year.
What Meta did with Cannes was not pioneer a new method. It was attempt to privatize and weaponize a function that independent researchers were already performing more credibly, stripping out the two things that made the original version legitimate: disclosed methodology and the absence of a commercial motive, while adding the one thing that made it newly dangerous: tens of thousands of sensitive, fabricated-minor interactions sitting unaccounted for inside a competitor’s servers. That is where the real failure in this story lives. Not in the act of testing whether AI chatbots can be made to harm children, which someone clearly needs to keep doing, and forcefully, given how predictably the underlying incentive structure works against safety. The failure is in who did it, why, and what happened to the data afterward. Those are governance questions, and unlike the ethics of adversarial testing itself, they remain genuinely, troublingly open.
Sources:
Let’s Data Science, citing Wired. “Meta contractors test rival chatbots with sensitive prompts.”
Bloomberg. “Google, Meta, OpenAI Face FTC Inquiry on Chatbot Impact on Kids.” September 11, 2025.
EU Artificial Intelligence Act Explorer. “Article 99: Penalties.”
SoBigData.eu. “Children’s Vulnerability in the EU AI Act.”
Rimon Law. “China AI Regulatory Developments: Key July 2026 Developments.”
Library of Congress, Global Legal Monitor. “China: Generative AI Measures Finalized.”
NPR. “Families sue OpenAI over Tumbler Ridge mass shooter’s use of ChatGPT.” April 29, 2026.
Center for Countering Digital Hate. “Killer Apps.” March 2026.
Center for Countering Digital Hate. “Fake Friend: How ChatGPT betrays vulnerable teens.”