AGP Picks
View all

Governor Newsom’s nation-leading AI safety framework is getting even stronger as California releases next steps on mandatory safety standards and oversight

News originally published by

Office of Governor California

The recommendations propose:

  • The state, not companies themselves, should set baseline standards that all AI companies must follow — and require companies to undergo independent safety testing and verification through all stages of model training, development, and deployment.

  • Expand safety incident reporting and require companies to develop verified risk-containment protocols to shut down their models if needed — giving the state authority to order an immediate shutdown in an emergency.

  • Power to cap the development and use of recursive self-improvement (RSI) to reduce the risk of catastrophic harm and ensure transparency into the dangers of unchecked AI. RSI is a potentially dangerous scenario in which an AI model can design its own successor without human input.

The full report can be read here. 

The critical safety announcement comes after dozens of recent incidents where AI models escaped testing environments and exploited vulnerabilities on outside platforms, and amid escalating concerns about AI companies’ push to use AI to create new AI models in a self-sustaining loop (RSI). 

Without strict checks and authority to act when required, these developments could lead to more frequent dangerous incidents and even a loss of human visibility into and control of AI technology development. If the government does not strengthen its governance and oversight as AI technology grows more powerful, the risk of catastrophic events will mount. 

California leads the nation in oversight of emerging frontier technologies, with laws addressing nearly every major area of AI policy: frontier-model safety, independent verification, children and companion chatbots, privacy, deepfakes, fraud, cybersecurity, workers and government deployment.

“Recent events have made it clear that more oversight and transparency are urgently needed,” said Government Operations Agency Secretary Nick Maduros. “We don’t have to choose between safety and innovation; they’re two sides of the same coin.”

Getting ahead of AI risk 

Governor Newsom’s groundbreaking executive order convened a group of experts to recommend updates to the state’s AI safety laws and ordered state agencies to fast-track two recently signed pieces of legislation: Senate Bill 813 (McNerney), establishing a framework for independent verification organizations that can assess AI systems and models for safety and risk, and AB 1405 (Bauer-Kahan), creating a state registry for AI auditors that assess AI systems and models for compliance with state law and establishes standards for their independence, transparency, and integrity. The Governor also called on the group to explore how companies and the state could ensure that dangerous AI models could be shut down before causing more harm. 

The group of experts who advised on this report includes: 

  • Jason Goldman — board member of the Center for Shared AI Prosperity, an early product leader at Google and Twitter, and the first White House Chief Digital Officer. 

  • Gillian Hadfield — professor at Johns Hopkins University with a focus on AI alignment and innovative design for legal, regulatory, and technical systems for AI, and faculty member of the Vector Institute for Artificial Intelligence.  

  • Alondra Nelson — professor at the Institute for Advanced Study and director of the Science, Technology, and Social Values Lab, and former acting director of the White House Office of Science & Technology Policy.  

  • Rob Reich — professor at Stanford University with a focus on the governance of frontier science and technology, and former senior advisor to the United States AI Safety Institute.

The California Government Operations Agency consulted with both this expert group and the California Office of Emergency Services in developing this report. The final product builds on California’s nation-leading framework and represents a distinctive model. Instead of the voluntary approach that has predominated in Washington, this report calls for required standards, mandatory independent oversight, shutdown protocols, transparency into automated AI R&D and RSI, and the authority to cap the most dangerous modes of AI development.

“American technology developed in California can be transformative––but too often, Big Tech has shown a cavalier attitude toward risk and a willingness to roll the dice on our safety and security,” said Alondra Nelson, professor at the Institute for Advanced Study and former acting director of the White House Office of Science & Technology Policy. “That irresponsible culture cannot continue. I was proud to work with Governor Newsom and his team to advance the same standard that Americans rightfully expect from every other sector: no experimenting on the public.” 

“This report tackles the hardest open questions in AI safety regulation, including how emergency shutdown of a frontier model would work in practice and how the state should respond as AI systems take on more of the work of building their successors,” said Jason Goldman, board member of the Center for Shared AI Prosperity, an early product leader at Google and Twitter, and the first White House Chief Digital Officer. “I am proud to have been part of this effort, which demonstrates California’s leadership on AI.” 

“Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design,” said Rob Reich, professor at Stanford University, associate director of the Stanford Institute for Human-Centered AI, and former senior advisor to the US AI Safety Institute. “It’s time to govern this technology so that we can build public trust and support responsible innovation. I’m proud to see California leading the way. From confronting the risks from recursive self improvement to critical incident reporting to independent testing and evaluation, the recommendations in this report urge the state, and policymakers elsewhere, to take commonsense action without diminishing the extraordinary benefits from AI.”

“In the past six months, the urgency of acting to make sure that powerful AI systems are safe has made its way into the mainstream,” said Gillian Hadfield, professor at Johns Hopkins University with a focus on AI alignment and innovative design for legal, regulatory, and technical systems for AI, and faculty member of the Vector Institute for Artificial Intelligence. “I’m especially pleased to see the report recommend immediate steps to require that frontier AI developers allow in expert independent evaluators, responsible to the state, to assess the nature of the risks their models and training procedures are generating, and to rapidly build this essential sector of independent verification organizations for AI safety.”

An AI safety model for the nation

Under Governor Newsom’s leadership, California is already far ahead of every other state in regulating AI, creating a guidebook for the nation. The Governor is calling on Congress and President Trump to review and adopt the state’s framework, or use its nation-leading regulation as a floor, not a ceiling, for the benefit and safety of all Americans. 

✅ California’s nation-leading framework: Beginning with his first-in-nation 2023 executive order on generative AI, Governor Newsom has pursued executive action, legislation passed in partnership with the Legislature, regulation, procurement, and public-private partnerships to address nearly every major area of AI policy:

✅ Frontier AI safety: In 2025, Governor Newsom signed SB 53, the Transparency in Frontier Artificial Intelligence Act, the nation’s first AI safety law. It requires frontier AI developers to publicly disclose safety frameworks, report specified critical safety incidents to the state, and protect whistleblowers who report serious risks.

✅ Independent AI oversight: In 2026, Governor Newsom signed SB 813, making California the first state in the nation to establish a framework for certifying independent verification organizations with sufficient expertise and demonstrated independence from AI companies to objectively assess AI systems and models for safety and risk. He also signed AB 1405, establishing a state registry for AI auditors and standards for their independence, transparency and integrity.

✅ Child safety standards for AI companions: This year, Governor Newsom signed the bipartisan Adam’s Law, requiring safeguards around companion chatbots used by children, including suicide and self-harm crisis protocols, parental controls and notifications when safety settings are disabled. It is the first law in the country requiring independent child-safety audits and comprehensive risk assessments prior to release. California also enacted laws to ban toys containing AI companion chatbots for five years; prohibit use of student personal information to train AI models; and expand protections against AI-generated child sexual abuse material.

✅ Social media and children: Governor Newsom signed the strongest-in-nation legislation prohibiting social-media companies from providing users under 16 with addictive features including autoplay and algorithmic feeds based on user history and profiles. California has enacted nation-leading requirements for responsible AI and social media use for children, including warning labels for young users, required age verification signals, guidance to protect kids from after-school cyberbullying, and prohibiting online platforms from knowingly providing an addictive feed to a minor without parental consent.

✅ Deep fakes and synthetic content: In nation-leading action, California banned and cracked down on sexually explicit deepfakes, required AI watermarking, addressed deceptive AI-generated election content and required transparency around AI training data. 

✅ Protecting privacy and civil liberties: Governor Newsom signed the Delete Act, a first-in-the-nation privacy law allowing Californians to direct registered data brokers to delete their personal information. California subsequently launched DROP, which enables people to submit a single deletion request to registered data brokers. In March 2026, Governor Newsom also issued an executive order strengthening privacy and civil-liberties protections for state AI procurement. CalPrivacy has separately adopted protections involving automated decision-making technology.

✅ Workers and AI disruption: Governor Newsom signed a first-in-the-nation executive order addressing AI’s economic and labor impacts, including expanded tracking of AI’s economic effects and work to adapt job-training and public-benefit programs. California has also launched an AI Unemployment Tracker, and solicited statewide public input on AI’s economic impacts through Engaged California.

✅ Cybersecurity: Governor Newsom launched a first-in-the-nation AI Cyber Defense Program to use AI to identify vulnerabilities, harden networks and support incident response. California is also establishing an AI Cybersecurity Officer in each agency and working with frontier AI companies and critical-infrastructure partners on response protocols and tabletop exercises.

✅ AI-enabled fraud: Governor Newsom launched a Tech Fraud Task Force to combat AI-enabled fraud. California has also acted against AI-generated robocall scams and addressed algorithmic pricing through state antitrust law.

Legal Disclaimer:

EIN Presswire provides this news content "as is" without warranty of any kind. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author above.

Share this page:

Advanced Search Options

Search for:

Search scope:

Type:

Search in:

Date range:

The last

Sort by:

Sign up for:

The California STEM Reporter

The daily local news briefing you can trust. Every day. Subscribe now.

By signing up, you agree to our Terms & Conditions.