ChatGPT and Anthropic could soon route extremist AI users to a new third-party tool: Report

People showing violent extremist tendencies on ChatGPT could soon be directed to human- and chatbot-based deradicalisation support through a new tool which is currently under development in New Zealand, according to a new report by Reuters. The tool is said to be aimed at addressing the growing safety concerns regarding the use of AI tools online.

Who is behind the tool?

Reportedly, the tool is being developed by ThroughLine, a startup that companies like OpenAI, Anthropic, and Google have previously hired to redirect users to crisis support when flagged for self-harm, domestic violence, or eating disorders.

ThroughLine founder Elliot Taylor told Reuters that the company is currently exploring ways to broaden its offerings to include preventing violent extremism.

The startup is also said to be in discussions with The Christchurch Call, an anti-extremism initiative formed after the 2019 New Zealand terrorist attacks. The group will reportedly provide guidance while ThroughLine develops the intervention chatbot.

“It’s something that we’d like to move toward and to do a better job of covering and then to be able to better support platforms,” Taylor said.

OpenAI has reportedly confirmed its relationship with ThroughLine, while Google and Anthropic did not respond to Reuters’ requests for comment.

How does the tool work?

The company is run from Taylor’s home in rural New Zealand and is said to have become the go-to for AI firms, with a constantly checked network of 1,600 helplines in 180 countries.

Once an AI detects signs of a potential crisis, it routes the user to ThroughLine, which matches them with an available human-run service nearby.

The anti-extremism tool would probably be a hybrid model combining a chatbot trained to respond to people who show signs of extremism and referrals to real-world mental health services, Taylor said.

For the anti-extremism tool, Taylor says the product would likely be a hybrid model combining a chatbot trained to respond to users showing signs of extremism and referrals to real-world mental health services.

Taylor says that the company will not be using “the training data of a base LLM”. Instead, he says it is working directly with “correct experts” to train the system.

He also says that follow-up features like possible alerts to authorities about dangerous users are still to be determined and would be decided after taking into account the risk of escalated behaviour. Taylor also noted that people in distress tend to share things online which they are embarrassed to say to a person. In this scenario, involving governments would only risk compounding the danger, he added.