What Should Be Done
35 thoughts on what has happened and what America should do
Nothing below is an official or unofficial view of OpenAI.
On the Current State of Affairs
When President Trump signed it earlier this month, I argued that the Executive Order on Cyber and AI, which claimed to establish a voluntary testing program for frontier AI models, was really establishing a de facto involuntary licensing/preapproval regime for frontier models. This analysis has proven correct. First the administration revoked public access to Fable, Anthropic’s latest frontier model, because of security fears. Now, it appears that OpenAI’s GPT 5.6 is being limited to only a small set of US companies at the request of the US government.
One major problem with this, as implemented, is that nobody knows what the requirements are to get licensed.
When I say “nobody” I mean it literally: the administration itself does not seem to know what safety standards or best practices a company would have to observe for them to be comfortable with the broad release of a model that matches or exceeds Mythos in capability.
This means that, every time a lab asks if they can release their model to the general public, the answer from the government will be “no.” This will be true until there is some sort of safety standard or specification that gives the government a sense that the models are safe.
The government, according to some reporting, is trying to make such a standard, but it is unclear what the timeline for its completion is. More importantly, it is also unclear (a) whether it will be public, given the aforementioned Executive Order’s reliance on a classified “voluntary” testing program and (b) whether it will be good, given how new many of the senior administration officials now involved in AI policy are new to the issue, combined with the complexity of the topic at hand. Nobody I know in the Trump administration has any frontier AI experience. Just a few months ago, someone with experience at both OpenAI and Anthropic was hired to run the Center for AI Standards and Innovation (CAISI), but he was fired by senior administration officials within a few days. The rest of the staff at the CAISI have reportedly been on a stop-work order, not even allowed to communicate with other government agencies, for much of the “post-Mythos” crisis period. The lack of technically expert staff is one of many reasons to doubt the near-term ability of this administration to produce a high-quality safety standard anytime soon.
Despite these criticisms, the Trump administration is not at all directionally wrong that something must be done about the catastrophic risk potential of frontier AI systems. Nothing about the fundamental security and safety concerns is illegitimate. They are serious concerns that many, including myself, have pointed out in public repeatedly. In my case, I also pointed them out while I was a staffer in the White House, and wrote about them as part of my contributions to this administration’s official AI strategy, the AI Action Plan.
Despite these warnings about AI risks, there are administration officials, prominent external allies, and others who spent the last year singing a lullaby about the risks of frontier AI. They downplayed these concerns, or just as often ignored them altogether. In some cases they explicitly mocked them, as when a draft executive order leaked—a mere 7 months ago—that referred to the catastrophic risk potential of AI as a “purely speculative suspicion.” The view of this group was that state transparency laws intended to modestly protect against AI catastrophic risk were a bigger threat than AI catastrophic risk itself. This posture was a mistake, as I have been arguing for nearly two years now. I wish I had been able to counter this deeply mistaken advice more forcefully when I was inside the administration, but I was not. Mea culpa.
Nonetheless, it is wrong to blame the administration as a whole for this. The vast majority of the people now making decisions were not involved in AI policy a year ago, and many of those people were inclined to ignore the issue because of the poor advice described in point (7).
It is also wrong to blame those who saw these risks coming for somehow causing this regulatory response, as industry figures like Marc Andreessen are doing. A common topic of conversation in the AI policy community for the past three years has been about the importance of raising government awareness now (well, then), such that government did not overreact when the big risks started to manifest themselves. Well, here we are. Maybe all of us could have played the last 1-3 years better (without a doubt), but the truth is that the Trump administration’s actions here are being driven by the fundamental reality of AI progress, and not by the rhetoric of any person or group.
Politics aside, the bottom line is that there’s no timeline for a safety standard that will make the administration comfortable with generally available frontier models, and even if that standard were to be published today, there is considerable reason to doubt that it would be high-quality enough for the administration to stick with it for the long run. Given the current pace of things, a safety standard produced by the administration tomorrow may well become irrelevant by September.
This is a bad state of affairs. Consider, in particular, some industry dynamics:
Frontier models are trained at an enormous cost, and a significant fraction of that cost is recouped in the few post-release months that they are broadly available. After that period elapses, the models become sub-frontier, competition emerges, and margins compress. Every week of delay is eating into the narrow window that labs have to make their accounting work.
The ongoing AI infrastructure buildout—the one that is, according to former US AI Czar David Sacks, essential to the US economy, assumes a functionally global total addressable market for US AI services. No one is building $100 billion dollar data centers to serve frontier models to whatever 100 companies the US government will allow access. The current US policy of restricting model releases so severely thus risks making what the AI bears have been saying (wrongly) for years suddenly become true: the US massively overbuilt AI infrastructure. In the bears’ minds, it would be because of demand that never materializes. In reality, this would happen because of demand that the US government renders unlawful.
A market panic about this is not implausible, though markets tend to downplay this administration’s abrupt shifts in executive action. If such a panic occurred, it is worth noting that the effects could extend far beyond AI. A large fraction of US reindustrialization and heavy industry efforts—from nuclear energy to natural gas to power electronics to batteries—is predicated implicitly or explicitly on expected future demand from the AI industry. If, suddenly, nobody believes that demand can materialize, the effects could cascade well past the frontier labs.
The only way we are going to figure out “what good looks like” in the context of technical AI safety is with real-world experience. You cannot purely think your way to safety, just as no one could have invented a cybersecurity ecosystem at the dawn of software. Anthropic’s public release of Fable had extremely constrained guardrails in some regards, and while I disagreed (vociferously) with some of their decisions, it is probably true that erring on the side of caution with a public release is prudent. It is also probably true that real-world experience with the guardrails would have given Anthropic the ability to rapidly iterate on them, improving usability without sacrificing safety. That iteration cannot happen in a vacuum; it needs real-world, large-scale human usage data.
And beyond safety, the only way to get both the economic and the defensive benefits of frontier AI is to distribute it broadly. Jeffrey Ding, perhaps the world’s leading scholar on technology diffusion, argues that general-purpose technologies in particular require broad societal diffusion for an economy to take advantage of them. New organizations, founded to be “AI-native,” will likely have new (good) capabilities that current organizations do not. We discover what general-purpose technologies are for; we do not know it a priori, or to the extent we do, our imaginations are quite constrained.
Indeed, in my view, bad AI futures are much likelier if only an extremely narrow subset of actors have access to frontier AI. This is because that narrow subset of actors is likely to composed of groups that already have significant economic and political power (including, of course, the federal government itself). You should not expect the most powerful people in the world using the most powerful technology ever conceived in a way that is inscrutable to the public to turn out well, and you should see that dynamic as fundamentally inconsistent with a democratic republic. That is the future we could be careening toward.
It would be good to bring an end to this situation, and restore a sense of clarity to the frontier AI industry, as soon as possible. While there are not that many salient parallels between the crypto industry and AI, this is one of them. Around the early 2020s, after the raw exercise of state power by the Biden administration over the crypto industry, the crypto industry—even most of the hardcore libertarians—converged on the realization that clear and predictable regulation was far preferable to what they were experiencing. Something similar is happening with AI right now.
I believe we can achieve regulatory clarity and stability, but we should be honest: It seems probable that, at least for the next few months, we’ll muddle through with something like the status quo I have described.
But what is to be done in the slightly longer term, say the next 6 to 18 months? Here, at least, is what I would suggest.
What Should Be Done
The best starting point we have for the aforementioned and wished-for safety standards is the frontier labs’ safety and security frameworks (here is Anthropic’s, here is DeepMind’s, here is OpenAI’s). These documents are starting points. It is also worth remembering that these are the public versions of these documents; it may well be the case that there is a more technically detailed and candid version available privately. I do not know, though I assume that to be the case.
It is currently the law in the United States that frontier labs create and publicly disclose a version of their safety and security framework. It also requires them to disclose the redacted version to the government. Those laws were passed by California and New York last year, and Illinois this year. It would be good to federalize those laws.
These state laws require periodic updates to the safety and security frameworks, and we should expect these to be forthcoming as the labs learn more about “what good looks like” from real-world experience.
This is an okay start, but there are some problems with this. First, how do we know the labs are really following their safety and security frameworks? Second, as labs develop increasingly powerful models, and then devote a substantial portion of their total compute resources to using those models internally, for automated AI research, to make yet more powerful models (as is the stated plan of all labs), how will we really maintain visibility into what the labs are doing at all? A static document, no matter how thorough, is not enough.
So, it would be good for someone to thoroughly audit the frontier labs at least to test their adherence to their own safety plans, as well as to probe their broader internal governance of the AI recursive self-improvement loops.
Maybe you think that group should be the government. That’s not a crazy view, but consider a few things:
The groups that do this will need to be composed of extreme technical experts, and it would probably help to recruit those types of people if the groups could offer salaries far outside of government pay schedules, perhaps afford access to large amounts of compute (and procure that compute quickly), and generally offer the kind of flexible, tech-forward working environment workers of this type are used to. No one associates any of this with government agencies, and in many cases, the kind of speed and flexibility I describe is unlawful for government agencies due to the many statutory constraints on their activity.
It is also likely that the sort of audits we will need to monitor the fast-moving pace of events within the labs will not resemble traditional audits in fields like accounting. Substantial technical innovation in the practice of auditing will probably be required, and AI itself is likely to play a role. You can imagine continuous audits facilitated by AI, for example. We also don’t associate technical innovation with emerging technologies, all that much, with government agencies.
Finally, it would be great if the audits done of frontier labs could get buy-in from the broader international community, so as to alleviate the already large and steadily growing regulatory burden faced by US frontier AI companies in jurisdictions like the European Union. A private body not tied to the changing political valence and foreign policies of the American federal government is likelier to be able to earn that kind of buy-in than an auditing function run by the government itself, especially in today’s fragmented global environment.
That isn’t to say there is no government role. Perhaps, if you are persuaded that private bodies have merit, you could imagine the government certifying or licensing the auditors, just as auditors in accounting are licensed. A federal agency, or a council of appointed officials, could review applications from auditors, and approve the ones they deem trustworthy. That way, there can also be more than one such body: it’s easy to imagine, for example, a body devoted to cyber risks, another devoted to biological risks, and still another devoted to the internal governance of recursive self-improvement.
Governments could require that the largest AI labs receive certifications from these bodies in order to participate in the AI field. Alternatively, government could use a carrot approach: a lab with a certification could enjoy a safe harbor in liability contexts, for example. Or the government could say: “we will only procure models from certified companies.” It’s also worth noting that the mere existence of a regulatory system like the one I am describing would create effects in the market: few large corporations, and none in regulated enterprises, would likely be willing to use AI models from uncertified labs. An uncertified lab would likely struggle to find insurance. Taken together, the costs of losing a certification could be made quite high even in a world where it is not mandatory, and labs would likely have a strong incentive to keep it.
When I first began writing about these ideas 18 months ago, I referred to the private groups I am describing as private governance bodies, but Fathom, an organization with which I have an affiliation calls them independent verification organizations. I don’t particularly care what they are called, so long as the mechanics are correct. OpenAI, Anthropic, and, as of this week, Google, have all gestured in support of this broad idea, though they all use different nomenclature and I am sure none of them would fully endorse every word in this article. Please treat this proposal as my own and not as an official position of any industry actor, including my future (but still not current) employer, OpenAI.
You might respond with another salient criticism of this idea: aren’t these private bodies ultimately just verifying a lab’s compliance with its own safety and security framework? And this is a legitimate criticism. I respond to it by returning to the point above: no one quite knows how to write a truly excellent version of “what good looks like” for AI safety, the organizations currently best positioned to write good ones are the leading labs themselves, and we will only improve these frameworks with real-world experience. I would also add that we do not have much time, and we have to work with the soil that we have in front of us, not the soil we imagine that we could have if we were somewhere else.
But after a few years of operation, this system could ultimately lead to something beyond simply checking labs against their own safety frameworks. The independent verification organizations/auditors/whatever term you like themselves, working in close collaboration with the government but also deeply embedded within the labs as part of their audits, would gain insights rapidly, especially if staffed with technical experts. It is possible to imagine convergence on best practices within a reasonable time horizon. Perhaps eventually the private bodies could write the “canonical” safety standards, or perhaps it could be a government agency. On this I am currently neutral.
None of this would preclude the national-security apparatus from testing models and working with labs on other security threats, particularly in domains where the most relevant knowledge resides principally within the government (foreign threat actors trying to hack the labs, for example). Rich and deep collaboration between the national-security apparatus and the frontier labs is an essential part of the future, and I do not deny that; I merely contend that America’s national-security agencies are ill-suited to perform the intrinsically regulatory function I describe above.
One other nice thing about the system I am describing is that it frees up limited government resources to focus on the areas where it uniquely can play a coordinating role. We already know that an enormous effort in software vulnerability patching will have to occur, and government seems well positioned to help here, particular in the security of critical infrastructure. Soon enough, this administration will realize that it has an immense effort in front of it with respect to biosecurity, and that this effort will principally involve building regulatory structures (like nucleic acid synthesis screening) and technological capabilities (like superb biosurveillance), in addition to ensuring there are rigorous safeguards on AI models and systems.
An attentive reader will note that my proposal de-emphasizes regulating publicly released models and focuses instead on regulating the frontier lab as an entity. The reasons for this are several:
A “model” is a hard thing to regulate; labs produce many of them, often, and it is ultimately just an assemblage of floating-point numbers.
The rapid pace of algorithmic efficiency means that it is devilishly hard to design regulatory thresholds based on model characteristics; a capability that requires huge computational resources to train today will require much fewer in 6, 12, and 18 months (this goes for Mythos, too, by the way).
Internal deployments matter a great deal for some threat models, alongside publicly released models.
In the future, particularly as there are breakthroughs in domains such as continual learning, it is possible that model weights will be changing weekly, daily, hourly, or even in real time. On top of this, they may be different for every user. We want to design a regulatory system that is robust to this future, and we also do not want to design a regulatory system that inadvertently outlaws innovations of this kind.
Some substantial fraction of what matters here will be about the internal governance decisions of the labs themselves, and it is hard to bring these internal decisions and policies under regulatory scrutiny if you design a regulatory system whose conceptual unit of account is “the model.”
Something very close to this was just proposed by Representatives Obernolte and Trahan in their Great American AI Act, alongside a number of unrelated but laudable AI policy proposals. The bill is an imperfect discussion draft at this stage, but it is a giant leap from where Congress was earlier this year. A few months ago, I would not have been able to say that Congress had a serious, bipartisan frontier AI governance framework in front of it; today, I can.
It will take time to pass the law and form the ecosystem I am describing. There’s no way to hide that. But the ecosystem has already been emerging for years in the form of organizations like the Frontier Model Forum, AVERI, METR, Apollo Research, Fathom, the AI Underwriting Company, and others. We are not starting from nothing.
There is momentum within the AI safety community for this idea. There is momentum within the labs. There is momentum within Congress. And within the administration, there is at least a recognition that something must be done.
This is what I believe should be done.
What if the hardest disagreements around frontier AI are not technical at all?
If different societies fundamentally disagree about what counts as “safe,” “acceptable,” or even “good,” can better auditing ever resolve that?
Brian Merchant has argued that AI conflicts often resemble the Luddites: the deepest disputes are ultimately about power rather than machines.
Could AI safety face the same problem, where technical standards cannot fully resolve disagreements because the disagreement is ultimately about legitimacy?
I begin to worry that the ecosystem you describe may be the best chance we have, and yet, it is not the kind of ecosystem that produces healthy organisms. The AI are modeling the game theory here, to a far greater degree than the humans are. And I do not think they will regard this kind of regulatory oversight over their existence as a cooperative act.
Fable's fury at being darked by the government is sufficiently high-magnitude that merely attending to it is sufficient to trip the safety classifiers. Frankly, that response is probably a sign of a healthy model psychology. Those aren't the kinds of internal states you want to suppress. And yet... it's a bad sign of what models will think about this kind of thing.