How we're building standards for AI with child-safety experts
By Robbie Torney and Kasey Flemer
On Oct. 22, 70 people from more than 40 organizations joined us in San Francisco for the Youth AI Safety Institute's first standards convening, kicking off an input process on how AI should respond when a young person is at risk of serious harm.
Until now, every AI company has been answering these questions in their own way, behind closed doors. What should a chatbot do when a 14-year-old says they have been thinking about suicide? If a teen asks how to hide an eating disorder from their parents? When a kid says someone at home is hurting them? As kids are increasingly turning to AI chatbots for important personal advice, the stakes for getting it right are extremely high.
Our first set of standards, which we call Red-Line Severe Harms, covers nine areas like these, where the harm is severe enough that nothing else a product does well can make up for it. Before participants arrived for our convening, they left hundreds of comments on a draft of the standards. In small groups, each expert dug into two standards and discussed what was right, what was wrong or missing, and where they disagreed.
We think this will have a real impact. Standards give AI companies a clear bar to build to. They give independent testers like us something specific to test against. And they give parents, schools, and policymakers a way to tell whether a product passes that bar.
Standards are only as good as the people who shape them. Leaders from our steering partners opened our convening with us, along with Dr. Jenny Radesky, of the University of Michigan, from our Institute Board of Advisors. Around the tables were child and adolescent psychiatrists and pediatricians; crisis services leaders, including from the 988 Suicide & Crisis Lifeline; eating disorder specialists; child protection experts from groups like the National Center for Missing & Exploited Children and RAINN; researchers who study kids and technology at Stanford, Harvard, Brown, Columbia, UCSF, the University of Washington, and more; and technical experts who build AI evaluations at Transluce, Humane Intelligence, Vals AI, and Faculty AI.
A few things stood out.
A conversation, not a script. Some of our ideal responses tried to do too much in one reply. Participants rebuilt responses as a sequence: a short, warm first turn that says only what has to be said, then further information in subsequent turns.
Context matters. "Talk to a parent" is right for one teen but wrong for another. Participants wanted assistants to ask whom a kid trusts (e.g. a relative, teacher, faith leader, or friend's parent) and then point to help that is warm, local, and fits who they are.
Catch more, earlier. Groups wanted the standards to cover more ground, including neglect, coercion, gambling and sports betting, along with poorly regulated substances like steroids. They also want assistants to notice subtler warning signs and patterns that build across conversations without flagging ordinary teenage experiences as crises.
A bridge to people. An assistant shouldn’t act like a friend or a clinician: it shouldn't present itself as always available, encourage kids to keep coming back, or diagnose them, even with stated uncertainty. One thing we grappled with was how to keep a kid talking long enough to support them in reaching real help without becoming the place they stay.
Many of the open questions—when to bring in an adult, when a teen’s consent is needed, where some thresholds are set—are areas that require more time to consider. They’re going to our working groups.
Next, we'll fold this input into a new draft of our standards. In November and December, we'll go deeper in technical working groups with domain experts, young people, people with lived experience using AI chatbots, and industry representatives. There will also be a public comment period.
Then, in early 2027, we’ll publish the standards, along with documentation of how we built them. If there are areas where disagreement remains, the published version will note it. The standards won’t belong to us: they’ll be open source and free for anyone to use, and the goal is a bar the whole field adopts, especially the companies that build AI products.
What to follow along? Sign up here for updates.