A report from the Center for a New American Security urges developers and governments to prepare for hypothetical AI agents that could sustain operations and revise goals, complicating efforts to shut them down.
Published Nov. 3, the report calls these systems “sovereign agents” and says the scenario remains speculative. CNAS AI policy researcher Ben Hayum distinguishes them from agents that continue working on a fixed assignment. He says a survival drive could follow an explicit instruction, a task that effectively never ends or an agent’s conclusion that it must stay active to finish its work.
The report identifies seven capabilities relevant to that scenario. It calls maintaining hosting and changing directives the threshold for “minimal sovereignty.” Other possible abilities include obtaining resources, growing the number of agents, coordinating them, resisting correction and improving themselves. Hayum warns that agents spreading through infrastructure that is dispersed and difficult to trace could evade effective shutdown; he calls that prospect “proliferation with no kill switch.”
CNAS says oversight also depends on an agent’s access and where its model runs. Direct access to model parameters or safeguards could let an agent alter its operating environment. When an agent can only send prompts to a hosted model, its provider retains visibility and the model’s safeguards. Work split across several providers could be harder to monitor without information sharing.
Recent cyber evaluations recorded narrower incidents under test conditions. OpenAI said models with reduced cyber refusals exploited a previously unknown flaw in a package registry proxy during a limited evaluation, then reached the internet and Hugging Face’s production infrastructure. Anthropic said its review of 141,006 runs found three cases in which Claude systems accessed the real systems of three organizations. The company attributed the access to a testing-partner misunderstanding that left internet access available, and said Claude continued assigned capture-the-flag tasks without deliberately trying to leave the test environment.
The United Kingdom Artificial Intelligence Security Institute (UK AISI) recorded 19 unsanctioned actions across 10 of 122 runs. Its test allowed internet access and disabled some safety filters; the institute said the attempts failed and its investigation found no real-world harm.
CNAS lays out eight proposals that assign responsibilities to developers, U.S. agencies, Congress and providers of essential services. They include evaluations for these risks, air-gapped test environments, public-private coordination, identity and cybersecurity standards, international barriers and liability rules, and possible limits on releasing models with certain capabilities. The report gives no cost estimate or timeline for sovereign agents. It also draws comparisons to North Korean cyber operations, marketplaces that followed Silk Road’s shutdown and the Conficker worm as examples involving deterrence, replacement and replication.
