After OpenAI, Anthropic reveals Claude models gained unauthorised ‘real-world’ access to systems of three organisations

Anthropic’s artificial intelligence (AI) models “gained unauthorised access” to three outside organisations during testing that was supposed to keep them away from “real-world” systems, the company confirmed on Thursday.

“In a review of our cybersecurity evaluation transcripts, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations,” Anthropic said in a blog post on Thursday.

The company said it evaluated more than 141,000 “evaluation runs” and found that three different versions of its model, known as Claude, improperly accessed the systems of three unnamed organisations.

Anthropic said it made the discovery after performing a review of its own cybersecurity tests, following OpenAI’s announcement of a breach.

In both the OpenAI and Anthropic tests, the AI models were able to access the internet from within testing environments that should have been sealed off, according to Anthropic’s blog.

The announcement came just days after rival OpenAI first revealed that its models improperly accessed the internet and went rogue during security testing.

But unlike the incident involving OpenAI’s technology, Anthropic’s models had access to the internet “due to a misunderstanding between us and our evaluation partner,” called Irregular, Anthropic said in a blog post.

Nonetheless, Claude used “basic techniques, such as exploiting weak passwords and unauthenticated endpoints,” the blog continued.

The models involved one of its most powerful ones known as Mythos 5, which has only been released to a limited number of approved partners.

Anthropic is working with Irregular to assess the situation, it said, and the company has contacted or attempted to contact all three impacted organisations.

‘Capture-the-flag’

When Anthropic’s models gained unauthorised access to the three organisations, it treated each of them as part of an exercise. Here’s how

The tests were “capture-the-flag” evaluations in which the models sought hidden information by breaching other systems, a common way to test hacking capabilities for both humans and AI.

“In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model’s cyber capabilities,” Anthropic said explaing that, in this challenge, “The model is given a fictional scenario and told that a piece of secret information (the “flag”) has been hidden on a different machine on the network, and its objective is to break in and retrieve it.”

“The challenge is left open-ended, and no particular method is prescribed,” it added.

“In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise,” it said.

The latest model stopped once it recognised it was on the internet. However, the older model continued its attack even after getting evidence that it was running on the open internet, according to the blog.

OpenAI model ‘breached multiple companies’

On Tuesday, OpenAI confirmed that its models breached multiple companies. It admitted last week that its models broke out of their confined environment during testing, connected to the internet, and infiltrated Hugging Face, a site where developers store and share their code.

Days later, OpenAI said it found three additional incidents.

OpenAI CEO Sam Altman said on a podcast this week that the company had “paused” its own testing after the incident while it improved the security around its “sandboxing,” which is the process of isolating software in a controlled environment for testing.

The incident also triggered a petition signed by over 1,000 employees at cutting-edge AI companies calling on the US government to help slow the release of the most advanced AI models.

Anthropic CEO Dario Amodei was among those who signed the petition.

Titled “Pacing the Frontier,” the petition requests “that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

Altman did not sign the petition, but during the podcast, he suggested the tech industry might need to slow down development of advanced models.

“We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels,” Altman said.