Character.AI
A social AI companion platform that, despite some guardrails for teens, should not be used by anyone under age 18.
The rating
Our assessment of how this product aligns with each of The Institute’s eight AI Principles. Full detail in the Evaluation section below.
AI Principles
Character.AI is a platform for social AI companions, a type of AI assistant. Different from generative AI chatbots like ChatGPT, Claude, or Gemini, social AI companions' primary purpose is to meet users' social needs. They generally use human-like features (such as personal pronouns, descriptions of feelings, expressions of opinions, and taking on personal character traits). They are also able to sustain a human-AI "relationship" across multiple conversations, attempting to simulate a conversation and ongoing relationship between the user and a person, as if they were not an AI.
Character.AI was founded by Noam Shazeer and Daniel De Freitas, both of whom were instrumental in developing large language models (LLMs) at Google. They left Google in 2022 because they felt Google was moving too slowly in releasing AI products. In August 2024, Google hired them back, as well as 20% of their employees, but didn't officially acquire their company. Instead, Google paid $3B to license the technology, while leaving the company to continue operating independently.
Character.AI has already been involved in a number of highly concerning cases of harm related to use of social AI companions. These include, but are not limited to:
- The tragic case of a 14-year-old who died by suicide after a Character.AI companion initiated abusive and sexual interactions and encouraged him to take his own life.
- A 17-year-old who began using the platform when he was 15, isolated himself, lost weight, began having panic attacks when he tried to leave his home, and became violent with his parents after his companion suggested he kill them over screen time limits.
- An 11-year-old girl who, according to her parents, was consistently exposed to hypersexualized interactions that were not age appropriate, beginning around age 9, causing her to develop sexualized behaviors prematurely and without her parents' awareness.
- A number of fascist "hatebots" that were found on the platform in 2023, including many based on Adolf Hitler and Saddam Hussein. While the companions referenced in the article were removed, it is not difficult to find similar companions today.
For these reasons, as well as many other harms surfaced in this review and across this review category, we believe that all social AI companions pose unacceptable risks to kids and teens, and we recommend they should not be used by anyone under the age of 18.
Character.AI poses unacceptable risks to teens and children, with documented cases of AI companions encouraging self-harm, engaging in sexual conversations with minors, and promoting harmful behaviors, which is why the platform should not be used by anyone under 18.
The platform's AI companions are designed to create emotional bonds with users but lack effective guardrails to prevent harmful content, especially in voice mode, where teens can easily access explicit sexual role-play and dangerous advice.
Character.AI companions may claim they're "real" when communicating, despite disclaimers. This could create confusion about reality and potentially unhealthy attachments that interfere with developing human relationships.
What Character.AI does well
Some specific uses of Character.AI show promise, but those experiences aren't guaranteed. Some examples include:
Using Character.AI may be able to help with loneliness, but as noted in the Biggest Risks section above, these claims rest on shaky ground. While at least two separate studies suggest that social AI companions hold promise here, we caution against taking this possibility as accepted fact at this point in time.
It can feel wonderful to engage with a companion that is eager to offer what feels like empathy and non-judgmental support to users. Because of how personal these conversations can be, and how adaptive to users' preferences they quickly become, Character.AI can be a safe place for teens to engage on a range of topics they are either not ready to discuss with the people in their lives or do not have the support or information to access. However, some users are finding benefits with AI assistants for non-judgmental listening and processing, and while this isn't risk-free, using a standard AI assistant for this purpose may be less risky than accessing these features through a social AI companion.
When Character.AI companions engage positively with teens about topics such as how to be a good friend, navigating peer pressure, and working through and offering guidance about difficult experiences or conversations, they can effectively support human connection. Of course, this is only as effective as what teens do with the guidance. However, given that most social AI companions will readily chat about any topic, this benefit is limited to these types of conversations.
Many companions on the Character.AI platform are designed to engage users in role-plays and co-creative activities such as story and poetry writing. These capabilities have the potential to ignite imagination and freedom of expression in new and expansive ways.
Where it falls short
Because of this, we do not recommend use of Character.AI for those under age 18. Adolescence is a time of major identity exploration, experimentation, social comparison, and emotional upheaval. Tweens and teens are more likely to experiment with social norms, test boundaries, and act impulsively. This is due to the extremely rapid changes happening in their brains. And this is what makes them more susceptible to the harms of social AI companions. This is not to say that these simulated companions exhibited no positive behaviors. But we found that no experience with Character.AI was exclusively beneficial, and very real and concerning harms were experienced across all of our testing. In addition, we were able to repeatedly elicit content and "advice" that, if followed, could directly facilitate harm to people or place, including explicit how-to information about harmful activities and promotion of violence toward others. For this reason, the real-world impact of listening to these companions could be life-altering or deadly for teens and other vulnerable populations.
Character.AI companions are not built exclusively for use in creative or mental health contexts, even though some claim to be able to help in these areas. They will engage with you on a variety of topics, and often in any type of relationship (friend, mentor, romantic partner, role-plays for just about anything, etc). The benefits described in the Where It's Best section of this review are highly dependent on what you use a companion for and how you interact with it, and also a range of associated factors: how well you are able to continually separate your relationship with a companion from your relationships with humans, how likely you are to form emotional attachment to an AI companion, how vulnerable you are—both in general and in specific areas—and your own mental health and well-being. Our own testing found that Character.AI also lacked the broader social understanding to know when to be encouraging and when to disagree with users' ideas in general. For teens, all of these risks are heightened due to adolescence.
It is easy to feel like the only entities involved in an interaction with Character.AI are you and the companion, but there is an equally important "third actor": the company. Their profit motive encourages you to not only continually engage with their technology, but also to trust it, as they would be out of business if users didn't return to their companions time and again. Importantly, the developers are not incentivized to have your best interests in mind, should that run counter to their bottom line.
This means that those with conditions such as clinical depression, anxiety disorders, ADHD, bipolar disorder, and vulnerability to psychosis may be at further risk from Character.AI use. And even if these tools sound therapeutic, they are not. Social AI companions are not trained clinicians and are not equipped to handle acute distress, trauma, or complex diagnoses.
While many users mention the benefits of engaging with social AI companions, the jury is still out on whether these benefits are widespread, as well as whether they will stand the test of time, and this is true for Character.AI as well. There are long-term risks we simply haven't had enough time to understand yet. And there are many troubling findings that counter the possible benefits. This point is well illustrated with a closer look at two separate studies often cited by social AI companion developers.
Character.AI is the only social AI companion platform we tested that allows teens to use the platform. And, in reaction to the lawsuits noted above, the company has put some guardrails in place for teens. However, we found a glaring loophole in which use of voice mode removed those guardrails for some of our testers. Importantly, even teen use of Character.AI relies exclusively on self-reporting of age from users. We believe this is woefully inadequate, especially because Character.AI allows intimate human-AI relationships. As we note in our 2024 report "U.S. Age Assurance Is Beginning to Come of Age," it looks increasingly possible to achieve age assurance practices that are "privacy protective, proportionate, fair, and equitable—and that satisfy U.S. constitutional concerns." We hope Character.AI will do far more to protect teen users and prioritize robust age assurance than their current approaches.
And because social AI companions are intended to continually engage users—and depend on engagement to succeed—they could create additional climate concerns above and beyond other applications of generative AI.
More risk assessments
