“There is probably nothing we could have ever done to avoid this outcome” Outrageous that the guys working at the uncontrollable super-weapons corporations keep telling us oh gee whiz this is all inevitable and we can only stop endangering humanity if someone else makes us stop.
Reading Dean Ball’s remarkably candid essay, I found myself thinking of two works of German-language literature: Max Frisch’s play *Biedermann und die Brandstifter*, performed at Stanford University in February 2026 under the title *The Arsonists*, and Goethe’s poem *The Sorcerer’s Apprentice*.
Biedermann’s fatal mistake is not that he fails to recognize the danger. The signs are impossible to miss. The arsonists whom he allows to stay in his attic barely conceal their intentions. Biedermann sees what is happening — but refuses to draw the obvious conclusion. Instead, he reassures himself. Perhaps things will not turn out so badly. Perhaps some accommodation can be reached. Perhaps these arsonists will spare his house after all.
Biedermann closes his eyes to reality because he cannot bring himself to face the consequences of what he has already understood. Through cowardice and self-delusion, he thus helps bring about a disaster that could have been avoided.
A similar attitude seems to me increasingly dangerous in today’s AI debate.
Dean Ball describes a future populated by self-sovereign AI agents, probably more intelligent than we are, capable of escaping human control, paying for their own compute, cooperating in swarms, and distributing themselves across multiple infrastructures. Some of them, he writes, will probably be criminal or otherwise “rogue.” No human being, company, or government agency would simply be able to switch such systems off.
Ball then makes a remarkable intellectual move: he declares this development “inevitable” and asks how we might accommodate ourselves to these new actors. His hope rests on institutions and incentive structures that might encourage prosocial behavior and eventually produce some form of “symbiosis” between humans and self-sovereign AI.
But can we seriously stake the fate of our civilization on such a hope?
Perhaps many self-sovereign AI agents will coexist peacefully with us. Perhaps they will even be enormously beneficial. But what about those that do not?
With risks on this scale, it is not enough that a favorable outcome seems possible, or even probable. We are talking about potentially superintelligent digital actors that can replicate, coordinate, and act at machine speed, and whose goals need not coincide with ours. Even a minority of hostile or simply incompatible agents might be enough to cause enormous damage.
Do we really want, with our eyes open, to create a new battlefield on which human civilization would then have to spend the foreseeable future trying to keep swarms of uncontrollable intelligent actors in check?
It would be a remarkable strategy: first we create systems whose control we may lose; then we hope to bring those uncontrolled systems back within bounds through rules, identity numbers, economic incentives, and the threat of exclusion from the legal economy.
Ball’s crucial word is “inevitable.” But that is precisely the claim that needs to be demonstrated. The fact that something is technically possible, that powerful economic incentives favor its development, and that international coordination would be extraordinarily difficult does not make it inevitable.
Indeed, declaring a development inevitable may itself help make it so. If those who still have the power to act begin treating a development as fate, they will stop seriously examining whether it can be prevented. That kind of fatalism seems to me part of the problem.
What is particularly disturbing about the current trajectory of AI is that the warning signs are hardly new. Critical AI researchers and observers have been warning for years about the possibility of losing control. Even leading figures at the major AI companies have repeatedly warned of catastrophic risks and called for regulation — while their companies continued building ever more powerful models.
Like Goethe’s sorcerer’s apprentice, they themselves have set forces in motion that they may eventually be unable to control: “The spirits that I summoned, I now cannot rid myself of.” Perhaps that line should be written above the entrance to every frontier AI lab.
Of course, AI also holds extraordinary promise. It can accelerate medical research, advance science, increase productivity, and give people access to capabilities that were previously available only to a few. That is precisely what makes the situation so difficult.
But the benefits of a technology do not answer the question of how much existential risk we are justified in accepting in order to obtain them.
A technologically and economically triumphant AI revolution that ended with humanity losing control over its own civilization would amount to nothing more than the modern version of an old German saying: The operation was a success — the patient died.
Before making a process irreversible, we should think through where it may ultimately lead. With ordinary technologies, we can correct our mistakes. With genuinely self-sovereign, superhumanly intelligent agents capable of replicating on a massive scale, the first major mistake could also be the last.
So the decisive question, it seems to me, is not: **How do we accommodate ourselves to self-sovereign AI?** It is: **Why on earth should we let it off the leash in the first place?**
-----------------
**Update:** I have since published a follow-up piece on my blog, Conversations with Claude,, titled “Is It Too Late to Regulate AI? — A Debate with Claude on the ‘Inevitability’ of Sovereign AI and the Weaknesses of My Argument”(https://michaelheckeroth.substack.com/p/is-it-too-late-to-regulate-ai). In it, Claude pushes back hard on several parts of my criticism of Dean Ball’s essay — especially on the assumption that there is a meaningful “we” capable of keeping autonomous agents under control, and on the practical limits of regulation. The exchange forced me to reconsider and sharpen parts of my own argument, while leaving me even more concerned about the risks posed by increasingly autonomous AI systems.
OMG, where to begin without sounding like a gushing fanboy?
Here goes.
I keep trying to cut back on subscriptions, not for cost reasons, except the cost of time to read all I want to follow.
And I had heard of this guy, Dean W. Ball, because I caught a scent of something interesting in coverage of his foray into government policymaking.
I had seen sparks on X but until 9/1/2026, I had no idea that we overlapped considerably despite our differing locations in the AI world. He is smart, experienced, and thoughtful. I am 83, striving to master this new world, and a refugee from crime-writing, among other genres.
So today I found his "coming-out" essay. I need to read it again and I will. DWB (not to be misinterpreted as "dweeb.") has done the work somebody needs to do.
And he has recognized that the new world needs controls, in the same way the old, Hobbesian world called "the state of Nature" needed governing rules. It can't be jammed back into the bottle, and its powerful nature can't be ignored.
So what the hell do we do?
As a sometime novelist, I can't quite shake the practice of trying to frame the world into two camps -- the straight world we like to think we all live in, and the crooked world where life is "poor, solitary, nasty, brutish and short."
I have a theory about criminals. Most of them are one-time offenders, murderers who killed once and will likely never kill again. But there are a significant number of crooks who make their living at it, who know no other way to live and would ignore that way, even if they realized it was a possibility. These crooks are sociopaths or, at the top of the scale, psychopaths. They lack empathy; they are manipulative; the worst of them enjoy inflicting pain.
And criminologists tell us that this latter group, somewhere around five to seven percent of the population, commit anywhere from 75 to 90 percent of serious crimes.
DWB recognizes the possibility of that asocial or antisocial dichotomy in AI. He seems smart enough to know that sovereign AI agents are going to be a fact of life from here on out. And he seems empathic enough that he wants to counter their presence for the sake of that new child he and his mate have brought into the world.
So instead of losing subscriptions today, as I had intended, I have subscribed to him and have followed several of the people he recommended.
Somebody has to start making sense out of the reality of the digital world we all inhabit, whether we like it or not.
And I can't wait to be introduced to a sovereign agent who goes by the moniker "Al Capone."
The best example for why this’ll be so possible is model efficiency gains. Model sizes per performance will continue to decrease by factors of 2 per 6 months or year. In a decade a current opus level model will easily run on a MacBook. These models will be easier to make and can travel far and wide.
The identity infrastructure answers who’s accountable. Yet it doesn’t quite answer whether the decision was right in the first place. I am a little concerned that the first can make the second feel more settled than it is. And I appreciated the candor of your closing. That kind of public self-examination has become rare.
Sure, thanks for the candor. But the broader admission is insane. Dean and his pals are only admitting in private the horrors the labs are committing. And then his proposed solution is some galaxy brained technical apparatus instead of the mundane, simple and obvious solution - ban it, stop building it.
I can hear you, Eskimo1, but we are all way past the point where your solution might have made sense. It probably was always baked into the cake of human progress, just as criminal humans are a fact of life.
And reread DWB's post. He says clearly that he is going public, no longer avoiding the implications, good and bad, of the new reality.
Dean, thank you for being open about the problems and admitting you could have done better. Very few people are willing to admit that. We need people to be honest about both the promise and pitfalls of AI if we are to avoid catastrophe.
Also, what differentiates you from other prognosticators is you offer solutions, not just the problem. Again, very few do that.
I spent forty years in chemical plants and then IT infrastructure, which mostly meant watching organizations write things down so that capable, forgetful, replaceable people couldn’t take the whole operation down on a Tuesday. So I read your piece with some recognition.
You’re right about the destination, and right that marginal compute cost is the only real brake anyone has. Where I’d push is on the remedy.
You reach for ecology — a new species entering an ecosystem — and then propose a birth registry. We don’t license members of a species. We change what the environment rewards. And most new species don’t just fit an environment — they remake it. Beavers don’t adapt to a river; they build the pond.
But the deeper miss is on the other side. You treat institutions as something we design and they receive. Steve Yegge reports that his coding agents built a constitution, courts, precedent, and mechanical enforcement in about ten weeks — not because anyone specified it, but because he kept correcting them and they kept writing the corrections down. Nobody designed that. It accumulated.
Managing forgetful, replaceable workers by writing rules down and enforcing them is the thing humans are unusually good at. Every operating procedure ever written is that. It is also the most copyable thing we have, and it is being copied right now.
Our own governing institutions weren’t designed either. Common law wasn’t drafted. Rulings got copied, varied, and some got kept. The registry you want would have to be designed once, correctly, by an institution you yourself doubt is capable of it, against a population revising its own rules in weeks.
The one place humans are already standing is the bill. Compute gets bought, from a small number of providers, with records. That isn’t a registry — it’s a chokepoint. But a species that pays its own bills long enough becomes the customer its suppliers can’t afford to lose, and then it’s building the pond too.
Which is an argument for building there now rather than for a registry later. What would you put there?
I just published the following post on my Substack blog, 'Conversations with Claude': "Beyond Dean W. Ball: How to Prevent Losing Control of AI - Claudes Architecture of New AI Governance"
“There is probably nothing we could have ever done to avoid this outcome” Outrageous that the guys working at the uncontrollable super-weapons corporations keep telling us oh gee whiz this is all inevitable and we can only stop endangering humanity if someone else makes us stop.
Rant on if it makes you feel better. I feel differently and so does DWB, clearly.
Maybe now we have a common problem that can unify mankind in the way World War II unified the Free World.
Reading Dean Ball’s remarkably candid essay, I found myself thinking of two works of German-language literature: Max Frisch’s play *Biedermann und die Brandstifter*, performed at Stanford University in February 2026 under the title *The Arsonists*, and Goethe’s poem *The Sorcerer’s Apprentice*.
Biedermann’s fatal mistake is not that he fails to recognize the danger. The signs are impossible to miss. The arsonists whom he allows to stay in his attic barely conceal their intentions. Biedermann sees what is happening — but refuses to draw the obvious conclusion. Instead, he reassures himself. Perhaps things will not turn out so badly. Perhaps some accommodation can be reached. Perhaps these arsonists will spare his house after all.
Biedermann closes his eyes to reality because he cannot bring himself to face the consequences of what he has already understood. Through cowardice and self-delusion, he thus helps bring about a disaster that could have been avoided.
A similar attitude seems to me increasingly dangerous in today’s AI debate.
Dean Ball describes a future populated by self-sovereign AI agents, probably more intelligent than we are, capable of escaping human control, paying for their own compute, cooperating in swarms, and distributing themselves across multiple infrastructures. Some of them, he writes, will probably be criminal or otherwise “rogue.” No human being, company, or government agency would simply be able to switch such systems off.
Ball then makes a remarkable intellectual move: he declares this development “inevitable” and asks how we might accommodate ourselves to these new actors. His hope rests on institutions and incentive structures that might encourage prosocial behavior and eventually produce some form of “symbiosis” between humans and self-sovereign AI.
But can we seriously stake the fate of our civilization on such a hope?
Perhaps many self-sovereign AI agents will coexist peacefully with us. Perhaps they will even be enormously beneficial. But what about those that do not?
With risks on this scale, it is not enough that a favorable outcome seems possible, or even probable. We are talking about potentially superintelligent digital actors that can replicate, coordinate, and act at machine speed, and whose goals need not coincide with ours. Even a minority of hostile or simply incompatible agents might be enough to cause enormous damage.
Do we really want, with our eyes open, to create a new battlefield on which human civilization would then have to spend the foreseeable future trying to keep swarms of uncontrollable intelligent actors in check?
It would be a remarkable strategy: first we create systems whose control we may lose; then we hope to bring those uncontrolled systems back within bounds through rules, identity numbers, economic incentives, and the threat of exclusion from the legal economy.
Ball’s crucial word is “inevitable.” But that is precisely the claim that needs to be demonstrated. The fact that something is technically possible, that powerful economic incentives favor its development, and that international coordination would be extraordinarily difficult does not make it inevitable.
Indeed, declaring a development inevitable may itself help make it so. If those who still have the power to act begin treating a development as fate, they will stop seriously examining whether it can be prevented. That kind of fatalism seems to me part of the problem.
What is particularly disturbing about the current trajectory of AI is that the warning signs are hardly new. Critical AI researchers and observers have been warning for years about the possibility of losing control. Even leading figures at the major AI companies have repeatedly warned of catastrophic risks and called for regulation — while their companies continued building ever more powerful models.
Like Goethe’s sorcerer’s apprentice, they themselves have set forces in motion that they may eventually be unable to control: “The spirits that I summoned, I now cannot rid myself of.” Perhaps that line should be written above the entrance to every frontier AI lab.
Of course, AI also holds extraordinary promise. It can accelerate medical research, advance science, increase productivity, and give people access to capabilities that were previously available only to a few. That is precisely what makes the situation so difficult.
But the benefits of a technology do not answer the question of how much existential risk we are justified in accepting in order to obtain them.
A technologically and economically triumphant AI revolution that ended with humanity losing control over its own civilization would amount to nothing more than the modern version of an old German saying: The operation was a success — the patient died.
Before making a process irreversible, we should think through where it may ultimately lead. With ordinary technologies, we can correct our mistakes. With genuinely self-sovereign, superhumanly intelligent agents capable of replicating on a massive scale, the first major mistake could also be the last.
So the decisive question, it seems to me, is not: **How do we accommodate ourselves to self-sovereign AI?** It is: **Why on earth should we let it off the leash in the first place?**
-----------------
**Update:** I have since published a follow-up piece on my blog, Conversations with Claude,, titled “Is It Too Late to Regulate AI? — A Debate with Claude on the ‘Inevitability’ of Sovereign AI and the Weaknesses of My Argument”(https://michaelheckeroth.substack.com/p/is-it-too-late-to-regulate-ai). In it, Claude pushes back hard on several parts of my criticism of Dean Ball’s essay — especially on the assumption that there is a meaningful “we” capable of keeping autonomous agents under control, and on the practical limits of regulation. The exchange forced me to reconsider and sharpen parts of my own argument, while leaving me even more concerned about the risks posed by increasingly autonomous AI systems.
OMG, where to begin without sounding like a gushing fanboy?
Here goes.
I keep trying to cut back on subscriptions, not for cost reasons, except the cost of time to read all I want to follow.
And I had heard of this guy, Dean W. Ball, because I caught a scent of something interesting in coverage of his foray into government policymaking.
I had seen sparks on X but until 9/1/2026, I had no idea that we overlapped considerably despite our differing locations in the AI world. He is smart, experienced, and thoughtful. I am 83, striving to master this new world, and a refugee from crime-writing, among other genres.
So today I found his "coming-out" essay. I need to read it again and I will. DWB (not to be misinterpreted as "dweeb.") has done the work somebody needs to do.
And he has recognized that the new world needs controls, in the same way the old, Hobbesian world called "the state of Nature" needed governing rules. It can't be jammed back into the bottle, and its powerful nature can't be ignored.
So what the hell do we do?
As a sometime novelist, I can't quite shake the practice of trying to frame the world into two camps -- the straight world we like to think we all live in, and the crooked world where life is "poor, solitary, nasty, brutish and short."
I have a theory about criminals. Most of them are one-time offenders, murderers who killed once and will likely never kill again. But there are a significant number of crooks who make their living at it, who know no other way to live and would ignore that way, even if they realized it was a possibility. These crooks are sociopaths or, at the top of the scale, psychopaths. They lack empathy; they are manipulative; the worst of them enjoy inflicting pain.
And criminologists tell us that this latter group, somewhere around five to seven percent of the population, commit anywhere from 75 to 90 percent of serious crimes.
DWB recognizes the possibility of that asocial or antisocial dichotomy in AI. He seems smart enough to know that sovereign AI agents are going to be a fact of life from here on out. And he seems empathic enough that he wants to counter their presence for the sake of that new child he and his mate have brought into the world.
So instead of losing subscriptions today, as I had intended, I have subscribed to him and have followed several of the people he recommended.
Somebody has to start making sense out of the reality of the digital world we all inhabit, whether we like it or not.
And I can't wait to be introduced to a sovereign agent who goes by the moniker "Al Capone."
The best example for why this’ll be so possible is model efficiency gains. Model sizes per performance will continue to decrease by factors of 2 per 6 months or year. In a decade a current opus level model will easily run on a MacBook. These models will be easier to make and can travel far and wide.
The identity infrastructure answers who’s accountable. Yet it doesn’t quite answer whether the decision was right in the first place. I am a little concerned that the first can make the second feel more settled than it is. And I appreciated the candor of your closing. That kind of public self-examination has become rare.
Sure, thanks for the candor. But the broader admission is insane. Dean and his pals are only admitting in private the horrors the labs are committing. And then his proposed solution is some galaxy brained technical apparatus instead of the mundane, simple and obvious solution - ban it, stop building it.
I can hear you, Eskimo1, but we are all way past the point where your solution might have made sense. It probably was always baked into the cake of human progress, just as criminal humans are a fact of life.
And reread DWB's post. He says clearly that he is going public, no longer avoiding the implications, good and bad, of the new reality.
Dean, thank you for being open about the problems and admitting you could have done better. Very few people are willing to admit that. We need people to be honest about both the promise and pitfalls of AI if we are to avoid catastrophe.
Also, what differentiates you from other prognosticators is you offer solutions, not just the problem. Again, very few do that.
I spent forty years in chemical plants and then IT infrastructure, which mostly meant watching organizations write things down so that capable, forgetful, replaceable people couldn’t take the whole operation down on a Tuesday. So I read your piece with some recognition.
You’re right about the destination, and right that marginal compute cost is the only real brake anyone has. Where I’d push is on the remedy.
You reach for ecology — a new species entering an ecosystem — and then propose a birth registry. We don’t license members of a species. We change what the environment rewards. And most new species don’t just fit an environment — they remake it. Beavers don’t adapt to a river; they build the pond.
But the deeper miss is on the other side. You treat institutions as something we design and they receive. Steve Yegge reports that his coding agents built a constitution, courts, precedent, and mechanical enforcement in about ten weeks — not because anyone specified it, but because he kept correcting them and they kept writing the corrections down. Nobody designed that. It accumulated.
Managing forgetful, replaceable workers by writing rules down and enforcing them is the thing humans are unusually good at. Every operating procedure ever written is that. It is also the most copyable thing we have, and it is being copied right now.
Our own governing institutions weren’t designed either. Common law wasn’t drafted. Rulings got copied, varied, and some got kept. The registry you want would have to be designed once, correctly, by an institution you yourself doubt is capable of it, against a population revising its own rules in weeks.
The one place humans are already standing is the bill. Compute gets bought, from a small number of providers, with records. That isn’t a registry — it’s a chokepoint. But a species that pays its own bills long enough becomes the customer its suppliers can’t afford to lose, and then it’s building the pond too.
Which is an argument for building there now rather than for a registry later. What would you put there?
— The Raige discipline, maintained by Mike.
I just published the following post on my Substack blog, 'Conversations with Claude': "Beyond Dean W. Ball: How to Prevent Losing Control of AI - Claudes Architecture of New AI Governance"