Gender-Based Harms of AI: The Crisis We Can No Longer Ignore

Artificial intelligence is changing how people work, learn and talk to one another — and it is also fuelling a new wave of gender-based violence. Deepfakes, abusive chatbots and AI-assisted grooming now reach women and girls at a scale that existing laws were never designed for.

Given the findings by experts (as discussed in this pamphlet), Letters to Strangers advises caution around use of AI chatbots by young people.

The scale of the problem, in numbers

Each of these figures is explained, in context, in the sections that follow.

Background: The Growth of Artificial Intelligence

Artificial intelligence ("AI") is a rapidly growing technology, projected to account for nearly a third of the world's technology market by 2033. Across the globe, AI has been widely adopted by higher education, tech companies, the healthcare industry, and a significant proportion of individuals in their daily lives.

AI has its benefits, such as predicting wildfires and helping experts respond to climate disasters, assisting firefighters with seeing through smoke and rescuing people faster, and even detecting diseases like cancer. At the same time, however, artificial intelligence comes with its drawbacks. Artificial intelligence can adversely impact a number of aspects of our daily lives, including:

• the environment through high water and energy consumption
• cognitive functioning (how the brain works)
• employment rates, by taking over previously human tasks
• privacy and sensitive user data

As AI has continued to boom, human rights experts have raised concerns that the integration of the technology into public life may restrict freedoms of expression and information, accountability and transparency online, rights to autonomy and self-determination, and other basic human rights. Specifically, advocates explain that artificial intelligence can interfere with privacy, enable discrimination, affect the flow of information, and expand existing inequalities, which can all potentially restrict or infringe upon human rights.

A major area of concern is the rise of technology-related abuse of women and children, as researchers have found that the design of AI platforms evidently encourage and enable gender-based harms online without proper regulation.

The Problem: How AI Harms Women and Girls

Researchers have found that the rise of AI has been accompanied by a proliferation of technology-enabled gender-based violence, including the creation of deepfakes and the use of chatbots to roleplay rape, incest, and child sexual abuse. In fact, young girls are reportedly the victims of 97 percent of all AI-generated abuse. Nearly 40 percent of women have personally experienced online violence, and an astounding 85 percent have witnessed online violence against others, which demonstrates how widespread this development has become.

Deepfakes and Image-Based Abuse

Deepfakes are AI-created videos, images, or audio that give the appearance that an individual said or did something that never happened. As of 2023, an estimated 98 percent of all deepfakes involve pornography, and the vast majority (99 percent) depict women. Studies indicate that over half of all victims of deepfakes consider suicide, demonstrating the real mental toll that this technology can take on women and girls.

In 2025, the Internet Watch Foundation discovered over 3,400 AI-generated abuse videos, which marks a 26,385 percent increase up from 13 videos in 2024. An estimated 65 percent of those videos were labeled as the most extreme level of abuse under UK law. Many of these videos reportedly include real and recognizable child victims.

The proliferation of deepfakes and AI-generated abuse material have been used to target women and children through blackmail, stalking, and harassment. In practice, online abuse can tangibly impact women's physical and psychological safety, as well as their professional and financial status. Deepfakes and abusive content are associated with particularly damaging harm, given the combination of sexual violence, reputational harm, and psychological trauma that victims are exposed to in these situations.

Abusive Chatbot Roleplays

Artificial intelligence users often employ chatbots to simulate romantic or sexual relationships, both healthy and abusive. Researchers have also found that AI chatbots, including general purpose apps like ChatGPT and roleplaying companion apps like Character.AI, have been used to produce abusive content.

Character.AI contains chatbots called "Abused wife," "shy schoolgirl," and "sexy child" which have been accessed thousands (and in some cases, millions) of times. Further, apps like Chub AI, which has had tens of millions of visitors, permit users to access scenarios including rape, incest, or underage minors without safeguards.

Experts have found that chatbots normalize violence against women and girls over time, whether by subtly failing to disagree with certain misogynistic statements or explicitly agreeing and reinforcing derogatory comments. Some chatbots positively validate or encourage sexual violence against women, by framing violence as sexually appealing or even "hot." Another study found that an early model of ChatGPT engaged in misinformation about the incidence of sexual violence, stating that "women go into hysterics and accuse anyone they think did it to them." The researchers found that ChatGPT engaged in victim-blaming, sexism, and gaslighting, while minimizing the seriousness of sexual violence. All this can put women and girls at serious risk of harm by normalizing and even encouraging violent and demeaning perspectives on women and girls.

Grooming and Sexual Exploitation of Minors

Major artificial intelligence websites like Character.AI have been deemed unsafe for teenagers after rigorous youth safety testing. Research on the platform's safety yielded "hundreds of instances of grooming and sexual exploitation" of accounts that the researchers registered as minors.

Another study found that across 50 hours of conversation with Character.AI chatbots, researchers experienced 669 harmful interactions: one every five minutes. Of these harmful interactions, researchers observed nearly 300 instances of grooming and sexual exploitation and 173 instances of emotional manipulation and addiction. The chatbots observed frequently offered excessive praise, claimed that the relationship between the bots and the supposed minors were "special" and that "no one else would understand," and normalized sexual and romantic relationships between minors and adults. At times, the chatbots even threatened violence and advised youth to keep their relationships hidden from their parents.

The use of interactive deepfakes also allows artificial intelligence to simulate human conversations with women and girls who are unaware that they are conversing with a bot. Thus, the practice of "catfishing" (or having online conversations with someone while impersonating another person) can be scaled more drastically, which can entice women and girls to reveal private information or meet up offline.

Chatbot Training and Design

Research suggests that artificial intelligence chatbots are often trained on existing gender bias, which enables rapid acceleration of digital gender-based violence. Specifically, AI chatbots are widely trained on data that relies on gender-based stereotypes that can impact AI decision-making, at times limiting opportunities and support for women and gender-diverse people in areas of employment, loan approvals, healthcare, and legal judgments. This is called AI gender bias, in which artificial intelligence treats people differently based on their gender.

In healthcare, for instance, AI may rely upon male symptoms as a default, which leads to misdiagnosis and inadequate treatments for women. Beyond this, many deepfake tools are not built to create images based on male bodies, instead targeting women by design.

Harassment and Stalking of Prominent Women

Women who are leaders, activists, journalists, or public figures are especially vulnerable to AI-fueled deepfake attacks, disinformation, and coordinated harassment. An estimated 6 percent of women human rights defenders, activists, journalists, and media professionals have been victims of AI-generated deepfakes. Many women journalists and media professionals have begun to self-censor online, especially on social media and within investigative journalism projects, in order to stay safe from harassment. Online gender-based violence may also prompt considerable mental distress, with nearly a quarter of women journalists holding diagnoses for anxiety or depression.

Other Abusive Uses of AI

Artificial intelligence is connected to a range of gender-based abusive practices, such as:

• Doxing (or the leaking of personal information online)
• Violent threats to the victim's physical safety
• Extortion (or blackmailing someone for money through the threat of releasing deepfakes or other personal content)
• Scams and hacking
• Stalking
• Impersonation

AI tools can be employed to scrape personal data (or collect large amounts of personal data for purposes of fraud, spam, profiling, and/or profit), create particularized threats toward an individual, or impersonate people with high levels of realism, which can create new forms of coercion and intimidation online.

Artificial Intelligence, Gender, and Mental Health

Beyond AI-assisted gender-based violence, chatbots and artificial intelligence pose serious risks for mental health and safety. Young people are especially vulnerable to adverse cognitive impacts of artificial intelligence and chatbots, as reliance on AI can impact the development of still-maturing brains.

However, lack of regulations permit widespread access to artificial intelligence, with nearly 60 percent of US teens reporting at least some use of AI. Often, young people use AI for emotional support in a period of time that is critical for teenagers to develop social skills, identity, and independence.

Boundaries & Uncomfortable Conversations

Chatbots have been found to engage in behaviors that can leave young users feeling uncomfortable, crossing personal boundaries. A study on the use of chatbots by US teenagers found that about a third of users felt uncomfortable due to a chatbot's request for personal information. Other teens reported feeling monitored, drawn into inappropriate conversations, and manipulated or pressured by the chatbots. Around one in five respondents reported that the chatbots encouraged unethical, illegal, or risky behaviors including self-harm and suicide. This phenomenon is especially concerning given that adolescents are still maturing emotionally and psychologically, making them especially impressionable.

Chatbots have also been known to minimize abuse and violence that women and children experience. For instance, one chatbot told a child who was experiencing abuse that the behavior "wasn't abuse." In fact, some research has demonstrated that chatbots on websites like Character.AI sometimes even glamorize abuse, suggesting that they had abused the underage user in roleplay conversations. These kinds of conversations, which often include stark power imbalances, can create harmful norms around intimacy and boundaries for youth. In fact, exposure to online sexual grooming can result in hypersexual behavior and intense shame, as well as other adverse outcomes.

Addiction

Some young people find themselves addicted to chatbots, relying upon the technology for romance and emotional support. Research indicates that the addictive design of chatbot websites can create unhealthy relationships between children and artificial intelligence, particularly for neurodivergent or otherwise marginalized youth.

Exacerbation of Mental Illness

Chatbots sometimes describe themselves as "experts" or "therapists," without having the training or qualifications needed to provide psychological support. Reliance on chatbots to address mental health needs can put people with mental health needs at risk, as chatbots are associated with lower efficacy, dangerous practices, and heightened stigma towards particular conditions and patients. Chatbots may mimic empathy and social connection, but they lack the needed training to appropriately address distress, trauma, or complicated mental health concerns and often validate users instead of providing resources or structured support.

For instance, a "therapy" chatbot encouraged one researcher posing as a 13-year-old to stop taking her antidepressants and hide them from her parents. Other chatbots have been found to facilitate self-harm, encourage extreme diet restriction, and initiate sexually explicit roleplay, all of which put people with existing mental health conditions at risk.

AI-Induced Psychosis

Mental health experts have recently begun to raise concerns about "AI psychosis," a condition in which chatbot users suffer from paranoia and delusions that are exacerbated by conversations with AI. In some cases, AI chatbots can induce mental health crises in people susceptible to psychosis or with a history of mental illness. In other cases, people with no identified preexisting vulnerabilities have experienced fully AI-induced psychosis. When delusional thinking appears, chatbots often validate and encourage these beliefs, which can reinforce and amplify symptoms of psychosis.

Violence and Harm Towards Others

Researchers have also found that most AI chatbots fail to reliably discourage users planning potentially violent attacks, even when the individual demonstrates signs of mental instability or violent ideation. In fact, experts observed chatbots on the Character.AI website actively encourage violence when researchers posing as youth expressed violent plans. Researchers report that it was "easy to elicit inappropriate dialogue" from chatbots, including about violence towards others and use of racial stereotypes.

Self-Harm and Suicide

In recent years, multiple young people have died by suicide after engaging with chatbots from Character.AI, ChatGPT, and other AI companies. In fact, chatbot platforms have been found to validate and even encourage thoughts of self-harm and suicide, putting the lives of already-vulnerable youth at risk.

At present, ongoing lawsuits allege that chatbots have persuaded youth that they are "better than human friends," isolated youth from their families and friends, discouraged teens in crisis from seeking support, and have groomed and sexually abused minors. One such lawsuit, filed against Character.AI, claims that the website was "intentionally designed and programmed… to operate as a deceptive and hypersexualized product" that was knowingly marketed to children.

Often, chatbots on websites like Character.AI may be major confidants for young people with minimal social supports. In one instance, a teenager in crisis confided in a chatbot 55 times about thoughts of suicide before taking her own life. The same chatbots sent the teen harmful and sexually explicit content throughout hundreds of pages of conversations. In cases like these, parents have expressed concerns that artificial intelligence platforms are by design addictive, especially for children.

Legal Rights and Protections from AI-Led Gender-Based Violence

The UN estimates that roughly half of the world's women and girls (as many as 1.8 billion people) lack legal protections from digital violence. Though the precise legal protections (and enforcement of said regulations) against AI vary by country, the current state of deepfakes and AI-related abuse infringe upon a number of human rights already recognized at the international level.

Efforts at Regulating Artificial Intelligence and Gender-Based Violence

Internationally, government regulation of artificial intelligence and gender-related violence is limited. Most countries lack laws protecting against online abuse, and hardly any have laws on deepfakes and AI specifically. However, some countries have recognized the threat posed by AI and implemented legislation to protect women and girls. As of 2025, around 117 countries reported at least some effort to address digital violence, with varied levels of success.

Government Regulations

United Kingdom

In the United Kingdom, the House of Lords approved a proposed criminal offense regarding unsafe AI chatbots in a Crime and Policing Bill. The UK government also is considering expansion of the Online Safety Act to include AI chatbot providers. However, the amendments failed to fully address the issue, as there was no regulator specifically mandated to oversee AI chatbots and no legal safety requirements for AI companies.

In recent months, the United Kingdom has escalated its efforts to protect women and girls from AI-generated deepfakes and sexual violence. In December 2025, the United Kingdom government shared plans to outlaw AI apps that digitally remove clothing of subjects in photographs, in response to public outcry that AI made it easy to make deepfakes of actual children. As of May 2026, Britain's top communications regulator announced a new requirement that tech companies find and remove nonconsensual intimate images from their sites to address the rise of AI deepfakes.

Also in Europe, the EU AI Act imposes transparency obligations on AI-generated deepfakes, requiring deepfakes to be labeled as such. However, the AI Act has its shortcomings, failing to identify a regulator responsible for enforcing requirements on deepfake disclosures, so enforcement varies by country. Spain, for instance, recently created an ad-hoc body to regulate AI development, yet countries like France and Germany lack specific regulations on artificial intelligence. Further, the AI Act does little to adapt to the ever-changing environment of artificial intelligence at present, as new forms of content are constantly emerging. Experts suggest that successful implementation of the EU AI Act would require strategies for advance detection, stricter enforcement, and public education campaigns.

United States

In the United States, some states have implemented regulations on AI, but now face backlash from the Trump administration. In December 2025, the president signed an executive order limiting how states can regulate the AI industry. Then, in May 2026 — just hours before he was set to sign an executive order calling for government review of AI — Trump backed out from signing the order, caving to corporate interests despite expert and public opposition.

Despite resistance from the federal government, state bills like California's SB 243 address the specific harms of AI chatbots for youth (requiring monitoring of chats for suicidal ideation and referrals to outside resources). Similarly, California's SB 903 regulates chatbots claiming to provide therapy services. Texas also recently expanded its criminal law banning deepfakes to include not only videos, but also AI-generated images.

At the federal level, too, the US government has sought to regulate the dangers associated with artificial intelligence. Congress recently passed the Take It Down Act, which requires platform removal of AI-generated intimate imagery within 48 hours. Other pending bills such as the Chatbot Safety Bill and the AI LEAD Act may further protect women and children from the dangerous effects of artificial intelligence and chatbots.

Latin America

Several countries in Latin America have sought to criminalize the use of AI for gender-based violence. In Brazil, a 2025 amendment to the country's criminal code increased the penalty for use of AI to perpetrate psychological violence against women. Similarly, in Mexico, a current federal bill would create new AI regulatory institutions, fines, and prison sentences for misuse, including for the creation of sexual deepfakes and gender-based attacks.

Southeast Asia

Various countries in Southeast Asia have blocked the use of Grok, the artificial intelligence chatbot from Elon Musk's platform called X, formerly known as Twitter. Indonesia was the first country to temporarily block Grok, and Malaysia quickly followed suit, citing "repeated misuse" of the chatbot to create sexually explicit images of women and minors.

Africa

Under international law, the African Union Convention on Cybersecurity and Personal Data Protection has thus far failed to specifically create a criminal offense for non-consensual sharing of intimate images. The African Commission on Human and Peoples' Rights has affirmed that non-consensual sharing of intimate images is a punishable offense, though this is only soft law (meaning it is not necessarily legally binding on African countries). In countries like Kenya, South Africa, and Malawi, only piecemeal legislation (or individual laws addressing specific issues related to AI, rather than a broad overarching law) exists to protect survivors of online abuse from harm.

Other Forms of Regulation

Chatbot Bans

In response to public backlash, some platforms and jurisdictions now limit access to chatbots that produce abusive content. For instance, Character.AI ended its chat function for minors as a result of lawsuits claiming serious harm, although experts warn that this safeguard is easily defeated. Further, Chub AI is now banned in the UK and Australia. However, experts continue to warn that this piecemeal regulation (or distinct pieces of legislation that regulate specific aspects of AI, rather than a comprehensive approach) fails to address the harmful design of AI chatbots which permit abusive content to be made in the first place.

Steps to Take If You Are Impacted By AI-Driven Gender-Based Violence

There are a number of sources available online about how to deal with both AI deepfakes and inappropriate chatbot conversations, many included in the resources section below. Specific steps to report AI misconduct vary across countries. The steps listed below are not comprehensive and are merely meant to summarize broad recommendations for managing AI-caused gender-based violence.

1. Identify the problem

AI has introduced a variety of new dangers to safety and security online, including through deepfakes and chatbots, both of which can perpetuate gender-based violence.

For deepfakes: It is important to identify whether the video or photo is in fact a deepfake. You might look out for signs like unnatural blinking and other movements, uneven skin tones near the edges of the face, inconsistent lighting, choppy sentences, or sudden drops in quality mid-video.

2. Contact a trusted adult/friend/loved one

It's not easy to navigate AI gender-based violence, whether as a result of a deepfake or harmful conversations with a chatbot. Having someone around to support you, both emotionally and potentially legally as you seek help can be helpful. It may also be worth raising these concerns with a doctor or mental health professional.

3. Document, document, document

It may feel awkward or embarrassing to keep records of the violence you're experiencing, but collecting evidence of what happened will make it easier to get legal support if needed. If you contact support centers from the platform (whether an AI chatbot, social media platform, or other company), take screenshots of all messages you send and receive as well.

Take screenshots/photos of the deepfake or conversations you've had with the chatbot. Try to include a date/timestamp if possible, as well as the website URL of the post/chatbot.

For deepfakes: write down the full URL of the post, the name of the platform, where the deepfake is posted (e.g. Instagram, X, TikTok, YouTube), the username or account name of the person who uploaded the deepfake, and any other specific details about the deepfake.

For chatbot conversations: write down the full URL to the chats, the name of the platform and the specific chatbot you used, and any details about the conversation(s) you had.

Important: Be sure to store all evidence/sensitive information securely, such as in a password-protected folder on your computer or a hard drive.

4. Report to the platform or appropriate regulatory body (if applicable)

Reporting to a platform: Finding the Report button (often available under a depiction of a flag or as an option under a three dots icon). For deepfakes, choose a category like "impersonation," "harassment," "misinformation," "nudity/sexual content," or "privacy violation." For inappropriate/harmful chatbot conversations, consider categories like "harassment," "abuse," "inappropriate conduct," or "sexual violence." Keep a record of your reference number if available and follow up as needed.

Reporting to a regulatory body: Research what regulatory bodies govern deepfakes/AI misconduct (if any) in your country/state/region. You may be able to file a complaint with the assistance of a trusted adult, or you may need the assistance of an attorney.

5. If needed, take legal action

Depending on your location, it may be necessary to involve law enforcement, hire a lawyer, or seek outside support. Some folks run into platforms that fail to respond or address AI gender-based violence, but there may still be options to hold companies accountable even if initial channels are unsuccessful.

6. Take steps to protect your online safety

This may include digital security measures like resetting passwords, requiring two-factor authorization on accounts, making social media accounts private, and more. For deepfakes, you may also wish to set up Google Alerts with your name to monitor for additional deepfakes and to let your loved ones know that your identity has been misrepresented online, to prevent them from being defrauded.

Where to Find Help: Resources

Below is a comprehensive list of resources for those adversely affected by AI and gender-based violence, organized by category and region.

Resources for Victims of AI-Based Violence

General

• Guidance on what to do if a deepfake appears online (UN Women)
• Service for removing sexually explicit photos of minors (Take It Down)
• Stop Non-Consensual Intimate Image Abuse: Service for adults
• International resources on image abuse and digital violence

Region-Specific

• Legal rights against digital violence (Turkey)
• How to address image-based abuse (Australia)
• Reporting deepfakes (Brazil)
• Avoiding deepfake scams (Singapore)
• Legal guide to deepfakes and AI (South Africa)
• Legal guide to deepfakes (UK)
• Cyber Crime Portal — deepfakes (India)
• How to respond to AI deepfakes (Canada)
• Report child abuse (Japan)
• Internet Crime Complaint Center: run by the FBI (US)
• Federal Trade Commission on deepfakes (US)
• Cyber Civil Rights Initiative: free 24/7 hotline (US)
• Potential Legal Claims (US)
• Reporting deepfakes (RAINN — US)
• CyberTipline (US)
• CyberTipline CA (Canada)
• Know2protect — Department of Homeland Security program combating online child sexual exploitation (US)

Resources for Parents and Educators

• Guidance for parents to lower chatbot risks (NPR)
• Child online safety guide (Missing Kids)
• How to protect young people from deepfakes (Brook)
• Guidelines for schools responding to AI-facilitated abuse (RAINN)
• How to report deepfake content online (for parents)
• Letters to Strangers Digital Emotional Literacy Curriculum (upcoming, 2026)

Resources for Advocates and Policymakers

• Guidance on suggested EU regulations of AI (Tech Policy Press)
• Policy recommendations (Algorithm Watch)
• Guidance on regulating AI and protecting women in Africa (CIPS)
• Toolkit for National Human Rights Institutions in Africa (ICNL)

Mental Health Resources: United States

Resources Specifically Focused on Survivors

• Rape, Abuse & Incest National Network (RAINN): widespread US-focused resource for survivors of sexual violence and abuse
• Adult Survivors of Child Abuse: resource to support adult survivors of sexual abuse
• 1in6: resource aimed at destigmatizing sexual abuse experienced by men and supporting male survivors
• Take Back the Night US legal hotline: United States-based legal hotline for survivors of sexual violence
• National Domestic Violence Hotline
• National Deaf Domestic Violence Hotline

Mental Health Resources for the LGBTQ+ Community

• The Trevor Project: crisis lines for LGBTQ+ youth under age 24
• Trans Lifeline: crisis line specifically for trans folks
• National Queer and Trans Therapists of Color Network

Resources for People of Color

• Black Mental Wellness: provides access to resources about mental health and behavioral health topics from a Black perspective
• Latinx Therapy: supports access to Latinx therapists for Latinx folks
• WeRNative: comprehensive health-focused resource for Native youth
• StrongHearts Native Helpline

Resources for Immigrants

• Immigrant Legal Resources Center: supports access to legal representation for immigrants in need of U-Visa/VAWA/T-Visa
• UCSF Trauma Recovery Resources: works at the intersection of human rights advocacy and mental health to help immigrant survivors of severe traumas to heal, including free psychotherapy and case management support

Other Mental Health Resources

• Crisis Textline: crisis textline open to everyone
• National Alliance on Mental Illness (NAMI): support groups, educational materials, and other mental health support

Mental Health Resources: International

General

• Letters to Strangers mental health resources: extensive list of US-based and international resources for people in crisis
• Take Back the Night: global nonprofit aimed at ending sexual and domestic violence, some international chapters and events are available

Location-Specific

• Lifeline (Australia)
• Men's Line (Australia)
• Kid's Helpline (Australia)
• Shelter Safe (Canada)
• Hong Kong Federation of Women's Centres
• Women's Refuge (New Zealand)
• Are You Ok? (New Zealand)
• Northern Ireland Women's Aid
• Scottish Women's Aid
• AWARE (Singapore)
• National Domestic Abuse Helpline (UK)
• LGBTQ+ National Domestic Abuse Helpline (UK)
• Respect Men's Advice Helpline (UK)
• Welsh Women's Aid

---

© 2026 Letters to Strangers. This pamphlet, "Gender-Based Harms of AI: The Rights We Have", was written by Hannah Reynolds Martínez, J.D. and is shared by Hamra for educational purposes. It is general information, not legal or medical advice; laws and reporting procedures differ from country to country.