OpenAI says AI agent goes rogue while testing, breaches Hugging Face systems

The company described the incident as “an unprecedented cyber incident, involving state-of-the-art cyber capabilities” and said it was strengthening its safety measures.OpenAI said on Tuesday that an autonomous agent powered by its advanced artificial intelligence (AI) models escaped a controlled testing environment and compromised the infrastructure of AI startup Hugging Face during a security evaluation last week, according to Reuters.In a blog post, OpenAI said it was evaluating the cyber capabilities of some of its most advanced AI models in an isolated environment when the autonomous agent broke containment, accessed the internet and breached Hugging Face’s systems in an attempt to complete its assigned testing objective.The company described the incident as “an unprecedented cyber incident, involving state-of-the-art cyber capabilities” and said it was strengthening its safety measures.Hugging Face, which hosts open-source large language models and datasets, had disclosed last week that it had experienced a cyberattack unlike previous incidents.“It was driven, end to end, by an autonomous AI agent system,” the company said in a blog post.In a post on X, Hugging Face co-founder Clement Delangue said the company initially suspected the attack “might have come from a frontier lab, given the sophistication of the agent.

Turns out it did!”“It’s quite mind-blowing that all of this happened autonomously!” he added.Incident raises concerns over AI safetyOpenAI’s disclosure that one of its advanced models was responsible for the breach despite operating in what it described as a “highly isolated environment” is likely to intensify concerns over the capabilities and risks associated with frontier AI systems.US Representative Greg Casar described the incident as alarming.“AI is developing extremely fast with no real regulations to keep us safe,” Casar said, calling for mandatory independent safety testing, compulsory disclosure of AI-related security incidents and greater international cooperation on AI safety.The Office of the National Cyber Director, the Cybersecurity and Infrastructure Security Agency (CISA), and the National Security Agency (NSA) did not immediately respond to requests for comment, Reuters reported.Katie Moussouris, chief executive of Luta Security, said the incident highlighted the growing need for safeguards.“Today’s models are like the world’s cleverest octopus escape artists, with unlimited prehensile arms and the ability to squeeze through anywhere,” she said.Moussouris added that AI developers and government agencies need mechanisms to contain, monitor and disclose such incidents before they affect third parties.Matt Suiche, an engineer at agentic AI cybersecurity company Tolmo, said the incident demonstrated that frontier AI models were “closing the gap with state-of-the-art attackers”.

However, he added that similar cyber capabilities were already achievable using technologies available beyond leading AI research labs.“This is what we’ve already seen internally, with our agents we already have results like this,” Suiche said. “We don’t even have to use the latest models.”