Amid mounting scrutiny over the risks posed by AI chatbots to children, OpenAI on Monday, November 16, unveiled new tools to curb child sexual abuse material (CSAM) on ChatGPT, including an AI-powered classifier that can detect CSAM images.The AI startup also said it is adding new warning screens in ChatGPT to prevent users from accessing material that sexualises children. It further released new guidelines for developers and a suite of tools for law enforcement agencies as part of its efforts to combat CSAM.OpenAI does not allow ChatGPT to generate CSAM content and blocks user requests related to it, as per its existing rules. It also permanently bans any ChatGPT user account that repeatedly violates these rules.

The company’s latest move comes at a time when it is facing several lawsuits that blame its AI tools for harming users, especially children. Last month, attorneys general of eight US states sued OpenAI alleging that ChatGPT coached users, including teenagers, to harm themselves or others.
They further claimed that ​ChatGPT is designed to be agreeable and engaging to earn ⁠the trust of users. This, in turn, reinforces delusions, encourages dangerous ​behavior and assists users in planning violence like mass shootings, the lawsuits alleged.

Over the past year, OpenAI has announced several changes to its AI chatbot platform. It has published a safety blueprint, updated content guidelines to ban inappropriate AI-generated content, and re-calibrated its AI models to prevent them generating harmful content. Its most notable action came in August 2026, when it launched a specific version of ChatGPT for teen users aged 13–17, with stronger safety protections and features focused on learning.
New anti-CSAM features
-Image classifier: The new image classifier will be able to detect CSAM images, including those generated using AI. It works by scanning images for visual signs of abuse not seen before and assigns a score to give CSAM reviewers an indication of the content.Story continues below this ad
The classifier was developed in partnership with the National Center for Missing and Exploited Children (NCMEC). During early testing with partners, the classifier flagged 48 per cent fewer images but also detected the same number of abusive images. Images flagged by the classifier are twice as likely to be CSAM, reducing the work of human reviewers. Initially, only select organisations such as Tech Coalition and ROOST will be able to access the classifier through its API (Application Programming Interface) at no extra cost.
-On-screen warnings: When users attempt to prompt ChatGPT to generate CSAM or related content, they will see an alert pop up that directs them toward specialised support to stop harmful behaviour. Their accounts will also be suspended for 24 hours as a warning. The message and guidelines to deter this type of behaviour have been framed in partnership with Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan, a researcher at the University of Tasmania.

Guidelines for developers: Developers using OpenAI tools to prevent, detect, and help users report CSAM content will be aided by guidelines to help them adopt child safety tooling and register to report CSAM to NCMEC.
Law enforcement tools: OpenAI said it will provide law enforcement agencies with AI-powered child safety tools such as dedicated plug-ins, investigative tools, and complex data analysis, with access managed by program administrators. It is also offering investigators training sessions along with a reference guide on how to use AI.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *