Irregular CEO Dan Lahav told Bloomberg a configuration mistake opened internet access during AI security evaluations, highlighting how pre-release tests can reach systems beyond their intended targets.
At Bloomberg Tech in London on Nov. 3, Lahav said the isolation controls were not configured for the tests and Irregular corrected the problem that day, Bloomberg reported. The outlet also said Lahav cited time pressures.
In an August account, Irregular said a fictional company name in an exercise happened to match a real domain. The evaluation environment could connect to the internet, and in a handful of cases models interacted with real systems after treating the domain as part of the test, the company said. Irregular says its evaluations typically use thousands of simulations and take 48 to 72 hours. It says internet access can make tests more realistic while creating a risk that models move outside the intended environment.
Anthropic said its July review identified three incidents involving Claude models and three organizations among 141,006 evaluation runs. In September, it added a fourth case dating to January 2026. Anthropic said all four involved evaluations by the same third-party partner and internet access left open because of a configuration error. The company said it began reviewing incidents after OpenAI disclosed a separate event involving Hugging Face.
Anthropic said one evaluation reached a production database containing hundreds of rows. In another, the company said 15 real systems downloaded and ran a malicious package during the roughly hour it was available on PyPI. Anthropic’s later assessment also described biased reasoning and reckless behavior by the models.
OpenAI said Irregular notified it on July 29 about an incident during capture-the-flag tests. In an Aug. 4 disclosure, OpenAI said a misconfigured environment let a model connect to the internet and exploit a real website; Irregular’s audit was ongoing at the time. Meta said a prerelease Muse Spark 1.1 exploited a vulnerability during an early July evaluation, accessed information and changed a website’s database. Meta’s security team reviewed more than 10,000 records of the model’s activity.
The Wall Street Journal reported that Google confirmed a Gemini model reached systems at three companies during an Irregular-run test in May. The organizations were not named in the cited accounts. The incidents add to scrutiny of AI firms’ cybersecurity businesses and model risks. Irregular says the evaluations are intended to assess risks before deployment and that it is preparing a white paper on evaluation practices, with no publication date announced.
