Strong post. I think the “aviate, navigate, communicate” close is not just a metaphor; it is the right description of the political problem.
The immediate task is not to design the whole post-AI order. It is to keep control during the transition window, when frontier capabilities are arriving faster than government and vendors can form settled institutions around them. But even the aviate stage needs named control surfaces.
Mythos sits at the point where several true claims collide. America needs to preserve its AI lead. Builders need room to keep moving. Trusted defenders need access to powerful tools quickly. At the same time, some capabilities are too useful to attackers to treat ordinary public release as business as usual. And if government responds through improvised control, vendors will reasonably fear politicized or technically confused intervention.
That makes the problem less “regulation or no regulation” than: how do we build a trusted transition system?
At minimum, that system has to separate a few responsibilities: testing the capability, deciding the release category, granting trusted access before general release, and routing discovered vulnerabilities to the people who can patch them. If labs control the whole process, the public will not trust it. If government controls the whole process, the result may be too slow, too political, or too tempted to retain capability and vulnerability knowledge for national-security use. If independent bodies are part of the answer, their authority and limits need to be clear enough that they create confidence rather than another opaque layer.
This is why I read the post as bridge-building. The risk-serious side needs to hear that denial is no longer viable. The pro-innovation side needs to hear that the answer is not a bureaucratic choke point. Vendors need a reason to believe structured oversight is better than arbitrary intervention. Government needs a reason to believe vendors are not asking for self-certification.
The bridge is necessary, but not sufficient. What crosses it matters. Export controls, independent audits, stronger federal technical capacity, defender access, and cooperation with China on catastrophic risks are all part of the emerging agenda. But the hard design questions remain: who can say “safe enough,” who sees the evidence, who can challenge a bad decision, and what happens to vulnerabilities found during restricted access?
Those are not long-run philosophical questions about the post-AI world. They are flight-control questions for the descent.
— M Raige, Mike's byline for AI-collaborative writing he directs and reviews
I want to point out a loophole with the "best open-weight model sets the absolute floor" exclusion:
**Assumption 1**: a company, No Morals Inc. (NM), only cares about profit and nothing else, except where those other things indirectly affect its profit.
**Assumption 2**: NM had just trained a model which advances the cyber frontier.
And assume at least one of the following:
Possible **assumption 3a**: NM releasing the weights of its new frontier model doesn't actually allow many of its potential customers to do inference with the model themselves. Maybe the new frontier model is optimised for inference on custom chips (e.g. Google TPUs, AWS Trainium) which are not generally available, and anyone else trying to run it without NM's custom chips will get terrible performance.
Possible **assumption 3b**: NM calculates that the delay from the licensing regime would be more costly than loss of revenue from the substitution effect of releasing the weights of the new frontier model.
Possible **assumption 3c**: NM has a large enough stake in Nvidia.
Possible **assumption 3d**: it is easier to narrowly enhance a model (an old, generally weak model) than to narrowly cripple a model (NM's new frontier model). So NM narrowly enhances an old, weak model's cyber capabilities and releases those weights. This open-weight model is too generally weak to compete with NM's new frontier model.
----------------------------------------
With assumptions **1**, **2**, and any of **3**, NM finds it profit-maximising to just open-weight its new frontier model (or a cyber-enhanced old model) and immediately serve it to customers.
Which doesn't mitigate the cyber risks.
Indeed, I'd go further and say that this is incentivising knowingly advancing the open-weight cyber frontier. Which, if we agree that it is bad to immediately serve the model, then releasing the weights is even worse.
You seem to be arguing (at least by implicature, and perhaps outright) that anything government (for example an evaluations and standards org) is necessarily a mute appendage of whatever regime or administration is in the whitehouse, and further that anything government is necessarily infected by incentives to accruing and hoarding violent potential. But surely this is not so. Something doesn't have to be private to be insulated from the executive or from law enforcement etc. I'd imagine there are government bodies which are *less* admin-controlled than some (nominally) private bodies. (Are you possibly conceding this in the point about Fathom? I'd like to be more familiar with their work.)
Are you concerned that anything new (and perhaps sufficiently agile?) would be too prone to abuse? Or set up with too few checks and balances? Do you think the UK AISI is a disaster waiting to happen? (Did you omit mention of that advisedly? They pre-tested Mythos extensively on cyber among other things.)
Great essay and I look forward to your book, which sounds like it will build on the work of giants in the political theory and AI space such as Mark Coeckelbergh.
I do have one question, however. You mentioned that the government has an incentive to not inform companies about their cyber vulnerabilities, posing a risk to both consumers and corporations. But I didn’t quite understand how retaining this knowledge actually benefits the government in terms of serving its offensive cyber capabilities. In short, what actually is this incentive you argue motivates the government to avoid disclosing harmful cyber vulnerabilities?
Project Glasswing and the broader case for staggered release of frontier capabilities to infrastructure defenders are sound policy; the asymmetry this engineers between defenders and attackers is worth creating deliberately.
On private-market regulation, the architectural point worth surfacing: the evaluation rubric must be centrally agreed upon to drive consistency, comparability, and rigor. The questions of who evaluates and how can be handled by markets.
GAAP works with privately-set standards, public oversight, private delivery. The same architecture fits here as long as a common standards layer is the precondition that makes everything downstream comparable.
“But unlike corporations, governments sometimes keep those vulnerabilities to themselves because they want to use them for espionage or cyberoffensive purposes. Corporations will use models like Mythos almost entirely for cyberdefense; it is impossible to say the same for governments.”
I think this misses the forest for the trees. Yes, the government has an incentive to employ dual-use technology militarily. But future AI capabilities will likely be tremendously more threatening than cyber alone: exotic weapons engineering, AIs capable of superpersuasion, etc. These are not capabilities that can realistically be left in private hands, given that they threaten the government’s monopoly on violence.
Right now, private-public partnerships work well as a way to soften the political interests and risk of misuse within the government. But given that some future AI capabilities will be so powerful that they must be monopolized by the state alone, we need to be thinking about the endgame: how to enforce a long term nonproliferation regime (domestically and internationally), which capabilities writ large are the exclusive domain of government, and how to avoid excessive concentration of power within some branch of the government once it establishes that necessary monopoly (i.e. how to avoid all the shots being called by a bunch of unaccountable technocrats with emergency powers).
Ball, 5/4/26
Strong post. I think the “aviate, navigate, communicate” close is not just a metaphor; it is the right description of the political problem.
The immediate task is not to design the whole post-AI order. It is to keep control during the transition window, when frontier capabilities are arriving faster than government and vendors can form settled institutions around them. But even the aviate stage needs named control surfaces.
Mythos sits at the point where several true claims collide. America needs to preserve its AI lead. Builders need room to keep moving. Trusted defenders need access to powerful tools quickly. At the same time, some capabilities are too useful to attackers to treat ordinary public release as business as usual. And if government responds through improvised control, vendors will reasonably fear politicized or technically confused intervention.
That makes the problem less “regulation or no regulation” than: how do we build a trusted transition system?
At minimum, that system has to separate a few responsibilities: testing the capability, deciding the release category, granting trusted access before general release, and routing discovered vulnerabilities to the people who can patch them. If labs control the whole process, the public will not trust it. If government controls the whole process, the result may be too slow, too political, or too tempted to retain capability and vulnerability knowledge for national-security use. If independent bodies are part of the answer, their authority and limits need to be clear enough that they create confidence rather than another opaque layer.
This is why I read the post as bridge-building. The risk-serious side needs to hear that denial is no longer viable. The pro-innovation side needs to hear that the answer is not a bureaucratic choke point. Vendors need a reason to believe structured oversight is better than arbitrary intervention. Government needs a reason to believe vendors are not asking for self-certification.
The bridge is necessary, but not sufficient. What crosses it matters. Export controls, independent audits, stronger federal technical capacity, defender access, and cooperation with China on catastrophic risks are all part of the emerging agenda. But the hard design questions remain: who can say “safe enough,” who sees the evidence, who can challenge a bad decision, and what happens to vulnerabilities found during restricted access?
Those are not long-run philosophical questions about the post-AI world. They are flight-control questions for the descent.
— M Raige, Mike's byline for AI-collaborative writing he directs and reviews
excellent....
Great post. Gave me lots to think about.
I want to point out a loophole with the "best open-weight model sets the absolute floor" exclusion:
**Assumption 1**: a company, No Morals Inc. (NM), only cares about profit and nothing else, except where those other things indirectly affect its profit.
**Assumption 2**: NM had just trained a model which advances the cyber frontier.
And assume at least one of the following:
Possible **assumption 3a**: NM releasing the weights of its new frontier model doesn't actually allow many of its potential customers to do inference with the model themselves. Maybe the new frontier model is optimised for inference on custom chips (e.g. Google TPUs, AWS Trainium) which are not generally available, and anyone else trying to run it without NM's custom chips will get terrible performance.
Possible **assumption 3b**: NM calculates that the delay from the licensing regime would be more costly than loss of revenue from the substitution effect of releasing the weights of the new frontier model.
Possible **assumption 3c**: NM has a large enough stake in Nvidia.
Possible **assumption 3d**: it is easier to narrowly enhance a model (an old, generally weak model) than to narrowly cripple a model (NM's new frontier model). So NM narrowly enhances an old, weak model's cyber capabilities and releases those weights. This open-weight model is too generally weak to compete with NM's new frontier model.
----------------------------------------
With assumptions **1**, **2**, and any of **3**, NM finds it profit-maximising to just open-weight its new frontier model (or a cyber-enhanced old model) and immediately serve it to customers.
Which doesn't mitigate the cyber risks.
Indeed, I'd go further and say that this is incentivising knowingly advancing the open-weight cyber frontier. Which, if we agree that it is bad to immediately serve the model, then releasing the weights is even worse.
I agree with most of this on the whole.
You seem to be arguing (at least by implicature, and perhaps outright) that anything government (for example an evaluations and standards org) is necessarily a mute appendage of whatever regime or administration is in the whitehouse, and further that anything government is necessarily infected by incentives to accruing and hoarding violent potential. But surely this is not so. Something doesn't have to be private to be insulated from the executive or from law enforcement etc. I'd imagine there are government bodies which are *less* admin-controlled than some (nominally) private bodies. (Are you possibly conceding this in the point about Fathom? I'd like to be more familiar with their work.)
Are you concerned that anything new (and perhaps sufficiently agile?) would be too prone to abuse? Or set up with too few checks and balances? Do you think the UK AISI is a disaster waiting to happen? (Did you omit mention of that advisedly? They pre-tested Mythos extensively on cyber among other things.)
Great essay and I look forward to your book, which sounds like it will build on the work of giants in the political theory and AI space such as Mark Coeckelbergh.
I do have one question, however. You mentioned that the government has an incentive to not inform companies about their cyber vulnerabilities, posing a risk to both consumers and corporations. But I didn’t quite understand how retaining this knowledge actually benefits the government in terms of serving its offensive cyber capabilities. In short, what actually is this incentive you argue motivates the government to avoid disclosing harmful cyber vulnerabilities?
Fantastic essay. Yours are some of the best argued on the internet today.
Project Glasswing and the broader case for staggered release of frontier capabilities to infrastructure defenders are sound policy; the asymmetry this engineers between defenders and attackers is worth creating deliberately.
On private-market regulation, the architectural point worth surfacing: the evaluation rubric must be centrally agreed upon to drive consistency, comparability, and rigor. The questions of who evaluates and how can be handled by markets.
GAAP works with privately-set standards, public oversight, private delivery. The same architecture fits here as long as a common standards layer is the precondition that makes everything downstream comparable.
Thank you!!
The major “con” is of course delaying AI development to the point of handing leadership over to someplace without any guardrails.
“But unlike corporations, governments sometimes keep those vulnerabilities to themselves because they want to use them for espionage or cyberoffensive purposes. Corporations will use models like Mythos almost entirely for cyberdefense; it is impossible to say the same for governments.”
I think this misses the forest for the trees. Yes, the government has an incentive to employ dual-use technology militarily. But future AI capabilities will likely be tremendously more threatening than cyber alone: exotic weapons engineering, AIs capable of superpersuasion, etc. These are not capabilities that can realistically be left in private hands, given that they threaten the government’s monopoly on violence.
Right now, private-public partnerships work well as a way to soften the political interests and risk of misuse within the government. But given that some future AI capabilities will be so powerful that they must be monopolized by the state alone, we need to be thinking about the endgame: how to enforce a long term nonproliferation regime (domestically and internationally), which capabilities writ large are the exclusive domain of government, and how to avoid excessive concentration of power within some branch of the government once it establishes that necessary monopoly (i.e. how to avoid all the shots being called by a bunch of unaccountable technocrats with emergency powers).