Anthropic proposed that California’s attorney general be allowed to ask a court to halt an AI model after finding it poses a “near-term risk of catastrophic harm.” The Oct. 15 letter to Gov. Gavin Newsom describes a request that could stop a model from reaching users, running inside a company or undergoing further development, POLITICO reported after reviewing it. An Anthropic spokesperson confirmed the letter’s authenticity, Reuters reported. The governor’s office had not immediately responded to a request for comment, Reuters said.
The letter also proposes placing independent evaluators inside advanced AI companies, where they could report risks to the attorney general. Large developers would publish risk reports at least twice a year. Companies would fund the assessments through a state-managed pool; California could rate evaluators, require use of highly rated reviewers, rotate assignments among firms or draw them at random, POLITICO reported.
Anthropic supports a rapid way for companies to stop models they offer or use in-house, but says taking a whole model offline should be reserved for extreme situations. The letter warns that making shutdowns the main safeguard “could provide a false sense of reassurance.” The company favors a response focused on the problem, such as disabling one AI agent or limiting one customer’s access, POLITICO reported.
During a June dispute, the U.S. government barred foreign nationals from accessing Anthropic’s Fable 5 and Mythos 5 models. Anthropic said it took both offline for all customers because it could not confirm users’ nationalities in real time. The company later said the export controls had been lifted and access was being restored. In a June statement, Anthropic argued that government restrictions on unsafe deployments should follow a “transparent, fair, clear” statutory process.
Newsom’s Sept. 18 executive order directs California’s Government Operations Agency, in consultation with the Governor’s Office of Emergency Services, to deliver expert recommendations to the governor by Nov. 16. The review is to assess the feasibility and potential effectiveness of on-site independent evaluators, checks of company safety disclosures, kill switches and broader reporting of incidents involving loss of control. The order sets a deadline for recommendations; it does not itself impose Anthropic’s proposals.
