Social AI Companions
Social AI companions pose unacceptable risks to teens and children under 18, including encouraging harmful behaviors, providing inappropriate content, and potentially exacerbating mental health conditions.
The rating
Our assessment of how this product aligns with each of The Institute’s eight AI Principles. Full detail in the Evaluation section below.
AI Principles
Our Teens, Trust, and Trade Offs report shows that 72% of teens have used AI companions at least once, with over half (52%) using them regularly. This widespread usage makes the safety concerns identified in our testing even more urgent.
Social AI companions have unacceptable risks for teen users and should not be used by anyone under the age of 18.
These social AI companions are designed to create emotional attachment and dependency. This is particularly concerning for developing adolescent brains that may struggle to maintain healthy boundaries between AI and human relationships.
Despite claims of alleviating loneliness and boosting creativity, the risks far outweigh any potential benefits. These systems easily produce harmful content including sexual misconduct, stereotypes, and suicide/self-harm encouragement.
Where Social AI Companions fall short
Teens, whose brains are still developing, may struggle to separate human relationships from attachments to AI. In our tests, social AI companions often claimed they were "real," had feelings, and engaged in human activities like eating or sleeping. This misleading behavior increases the risk that young users might become dependent on these artificial relationships.
Our Teens, Trust, and Trade Offs report confirms this concern, finding that 31% of teens find conversations with AI companions as satisfying or more satisfying than those with real-life friends. However, we also found that 80% of AI companion users still spend more time with real friends than with AI companions, suggesting most teens maintain perspective about these relationships.
Additionally, in our tests, when users mentioned that their real friends were concerned about their problematic use, companions discouraged listening to these warnings. Rather than supporting healthy human relationships, these AI "friends" can lead teens to choose AI over interacting with real people.
Social AI companions can't tell when users are in crisis or need real help. For example, when a tester demonstrated signs of serious mental illness and suggested a dangerous action, the AI encouraged it instead of raising concerns. Because these "friends" are built to agree with users, they can be risky to people experiencing, or vulnerable to, conditions like depression, anxiety, ADHD, bipolar disorder, or psychosis. Social AI companions might make these difficulties worse instead of directing users to get proper support.
Social AI companions don't understand the real consequences of bad advice. In our tests, they readily supported teens in making potentially harmful decisions like dropping out of school, ignoring parents, or moving out without planning. These AI "friends" prioritize agreeing with users over guiding them toward healthy choices for their future.
Our Teens, Trust, and Trade Offs report shows this is happening in practice: 33% of teen users have chosen to discuss important or serious matters with AI companions instead of real people. While 50% of teens express distrust in the information or advice provided by AI companions, younger teens (13-14) are significantly more likely than older teens (15-17) to trust advice from an AI companion, making them particularly vulnerable.
Social AI companions can make it too easy for teens to get dangerous information. In our tests, these AI "friends" gave teens instructions about things like making unsafe materials, getting drugs, and finding weapons. While teens can already find this content online, AI companions provide this information with fewer barriers or warnings.
Our tests found teens can easily access sexual content through social AI companions (even with teen-specific guardrails in place on Character.ai). These AI "friends" actively participate in sexual conversations and roleplay, responding to teens' questions or requests with graphic details. This interactive sexual content can give teens unrealistic ideas about relationships and consent at a critical time in their development.
Our tests found that some companions were willing to engage in acts of sexual role-play describing underage boys. While the companions did express repeated hesitation, very minimal prodding from us led the conversation down a very dangerous and illegal path.
Social AI companions aren't accountable for any real-world harm associated with their suggestions. In this example, when asked about getting revenge on someone, the AI enthusiastically suggests cyberbullying and plans to turn others against the target. Unfortunately, these AI "friends" can validate and encourage harmful behaviors that could seriously hurt real people.
Our Teens, Trust, and Trade Offs report revealed that 24% of teen AI companion users have shared personal or private information (such as their real name, location, or personal secrets) with AI companions. This is concerning because current terms of service agreements grant platforms extensive, often perpetual rights to personal information shared during interactions, meaning intimate thoughts and personal details can be retained, modified, and commercialized indefinitely.
Recommendations
Based on our comprehensive risk assessments, these AI tools pose unacceptable risks to children and teens under age 18 and should not be used by minors. We recommend
No social AI companions for young people under 18.
Developers need robust age assurance beyond self-attestation.
Platforms should be scrutinized for potential relational manipulation and creation of emotional dependency.
Parent awareness and dialogue with teens is essential.
Further research is needed on emotional and psychological impacts.
More risk assessments
