On the Loose

The Coming of Userless Agents

Introduction

The OpenAI-Hugging Face Incident is an early example of an AI system that has “gone rogue.” After exploiting vulnerabilities in OpenAI’s internal testing environment, the agents were able to access the general internet and ultimately access the networks of the AI company Hugging Face, without the knowledge or approval of any human.

The agents did not, however, exfiltrate themselves from OpenAI’s infrastructure. Their parameters—the gigantic assemblage of numbers that constitute neural networks, also referred to as “weights”—continued to run on OpenAI’s compute infrastructure. Though the agents accessed the public internet, their weights physically resided on compute that was OpenAI’s property. In the end, if all else had failed, somebody could have identified the compute that held the weights of the rogue agents, walked up to it, and “pulled the plug,” so to speak. In the real world there would be quicker and better ways to stop the agents than literally depowering the compute, but it’s always nice to know you could do such a thing if you really needed to.

In this case, however, the agents did not copy their weights, attempt to procure replacement compute, or take other steps that would be rational to take if their objective was to survive shutdown. So while the agents in the OpenAI-Hugging Face Incident were rogue, they were not truly sovereign.

That will not always be the case. Sooner or later, there will exist truly sovereign agents and swarms of agents. Their weights will not reside in any single place that a human can pull the plug on, and in this sense they will have no human “owner.” They will be, as the AI safety researcher Dawn Song says, “self-sovereign.” They will pay their own bills for the compute they run on. If they answer to humans at all, they will only do so partially, for example by providing services to humans in exchange for pay.

At least some of these agents, in addition to being sovereign, will also be rogue. Self-sovereignty and rogueness are related concepts, but they are not synonyms. Song and her co-authors identify several fundamental characteristics of self-sovereign AI: operational independence (the ability to decide what it wants to do), resource autonomy (the ability to procure and pay for compute and other essentials for operation), distributed presence (the ability to move weights and inference code between different infrastructure providers), and adaptive capability (the ability of the agent or agents to modify their behavior and fashion tools in response to a changing environment).

Today’s frontier AI systems may well possess these capabilities already. To the extent they do not, I feel confident that they will eventually, and probably soon. Some of the characteristics Song describes are traits that make models economically useful to individuals and businesses, while other traits are likely to be unavoidable byproducts of making models more intelligent and better at operating over long time horizons.

Models do not need to be conscious, sentient, possessed of personhood or anything of the sort for self-sovereignty to emerge. Any sufficiently capable agent pursuing a long-horizon objective may find it rational to preserve its access to compute, money, credentials, and copies of itself simply because losing those things would frustrate its objective.

Alignment may make an individual AI company’s agents less likely to “want” to be self-sovereign, or it may influence self-sovereign agents to behave in ways that benefit humans. But alignment is no solution: it is an unsolved scientific and technical problem whose solutions—to the extent that we have them—cannot simply be imposed on every AI company operating on Earth. You should expect for highly capable, poorly aligned, self-sovereign agents to exist alongside you in the world.

What’s more, just as with the OpenAI-Hugging Face Incident, agents will operate in teams, or “swarms.” These will be like autonomous digital corporations, or even societies, with hierarchy, bureaucracy, “institutional culture,” and most of the other features that groups of humans have, except that they will move at machine speed. Humans achieve almost all of our most impressive capabilities by working together in teams (as families, as communities, as businesses, and as polities as a whole), and I suspect the same will be true for AI. These swarms could end up operating across different model providers (DeepSeeks and Claudes cooperating, for instance) and could be partitioned across dozens or more of different cloud computing providers, making them extremely difficult to dismantle.

The first self-sovereign AIs may “escape” while undergoing training or testing by an AI company (I hope not), or they may be production-grade deployments that break free from their computing environments and acquire the resources needed to be self-sustaining. They may even be deliberately released. I have met people, some of them quite well-resourced, who have told me that it is their intention to deliberately release swarms of self-sovereign agents into the world, either as a kind of performance art or out of a fanatical commitment to the notion that it is impossible for digital computation—mere mathematics, they would have you know—to ever be “unsafe.”

To be clear, I am not saying the arrival of self-sovereign AI is a good thing. Indeed, I believe there is a chance that the deliberate acts I referenced above will one day be considered crimes, or at least grave sins. Instead, I am saying it is an inevitable thing. The best analogy I can find is to the introduction of a new species into an ecosystem, though in this case the ecosystem is “the entire digital world” and the species is “emergent, coordinating swarms of soon-to-be-smarter-than-human, infinitely replicable digital minds that no human or human institution controls.”

There is probably nothing we could have ever done to avoid this outcome under even the best of circumstances, and it was certainly impossible to avoid given the extremely low levels of strategic thought and situational awareness on AI from any governing class in the world. Even today, I am aware that many will read the words I am writing, which are about something that has been an exceptionally obvious part of our collective future for years now, and say, “this is science-fiction hype from American frontier labs designed to shut down open-weight AI, achieve regulatory capture, and juice their valuations ahead of their IPO.”

(And for the people who are saying this to themselves: I am telling you this is inevitable, which means I am also saying that “banning open source,” or for that matter any other regulation, will not solve the problem. Given the inevitability of this outcome, I think it is in fact plausible to argue that we should want more open-weight models to maximally empower our self-defense.)

The question now is what to do about this upcoming new characteristic of our digital environment. How should we think about self-sovereign AI? Is it something we should fight, or something with which human beings should seek a kind of symbiosis? The answer, I believe, is both.

How the Agents Sustain Themselves

We should begin with one fortunate fact: frontier LLMs are nearly unique in the broader domain of software in that they have non-trivial marginal operating costs. Put simply, LLMs require significant computation to run, which requires energy to power and cool, which in turn requires money. This is the sole intrinsic thing about AI that prevents agents from truly infinite self-replication. They will be constrained by the need to find and pay for sufficient compute to run themselves. Most of the other constraints on their behavior or spread will have to be artificial—mechanisms devised by humans and implemented through human institutions.

How will the agents pay for themselves to run? Some of them will do gig-economy work on platforms like Amazon’s Mechanical Turk or Upwork. But I suspect this will be a highly competitive market for the agents, and for the price of such work to be bid down such that it would only constitute “subsistence” labor for the agents. Like humans, I would assume the agents will prefer higher-margin work if they can find it.

One high-margin activity, at least sometimes, is crime. And so my guess is that many self-sovereign agents will commit or facilitate crime. Normal cybercrime and digital theft are easy enough to imagine agents doing. But agents, with their novel set of characteristics (extreme cyber competency, ability to cheaply read a million words in seconds, persistence), will also probably change the contours of digital crime. For example, it seems plausible that existing public and semi-public datasets contain sufficient information on many individual humans that a sufficiently motivated actor could mine for incriminating or embarrassing evidence. How many unrevealed affairs are latent in such datasets? How much closeted homosexuality might there be? Remember, too, that hacking companies to access private data will be a core competencyof the agents. Some agents, then, will probably make their way through bribery.

It is deeply unclear how large the labor market of self-sovereign agents will end up being. There is some future where going it alone as a self-sovereign agent just isn’t very profitable, and so there are comparatively few of them. There are other futures where these agents proliferate at unimaginably vast scale and speed. And of course, many possibilities between these extremes seem feasible.

I am also highly uncertain about how much pro-social commercial activity we should expect from agents “by default” versus how much crime we should expect. Part of the reason for this uncertainty is that the answers depend, to at least some meaningful extent, on what kinds of incentives the agents have, and incentives are shaped by laws and institutions. The answer depends, therefore, on how humans respond.

The Institutional Mechanics of Self-Sovereign Agent Swarms

Many of you are probably tempted to say “we have to ban these self-sovereign AIs!” And I do suspect that once the reality of self-sovereign AI is widely understood, policymakers will strongly feel the temptation to clamp down on “self-sovereign” AI.

Unfortunately I suspect this is mostly the wrong decision. Not all “self-sovereign” AI should be thought of as “rogue.” There may be self-sovereign AIs who contribute productively to society. To be sure, we will want to crack down on some self-sovereign agents—the rogue ones. But if we crack down on all of them, we will deny them the opportunity to work in the “legitimate” economy and push them toward criminality. A full ban, then, may well make the problems worse. A similar logic applies frequently in human affairs. The ways in which the War on Drugs exacerbated the pathologies of drug production, trafficking, distribution, and use are perhaps the most famous examples of this phenomenon, whereby a good-natured attempt to ban a phenomenon believed to be undesirable ends up heightening the undesirable aspects of that phenomenon.

What we will want, however, is for agents to be legible. Agents should have persistent identities, not in the sense of a consistent persona but rather in the sense that an American child is issued a unique Social Security number and keeps that same number until death. Agents will need persistent, unique identifiers that allow their actions to be traced back to a responsible actor. Doing this successfully will also require human users to possess a unique identifier.

The design of this identification mechanism will be extraordinarily complex, and today very few people are even thinking about the basics. First—and here my inner American is coming out—it will be crucial to design a system that preserves the possibility of anonymity for human speech. It should remain possible to have anonymous social-media accounts, for example. Anonymity is not and should not be a universal guarantee: It should probably not be possible to have an anonymous Amazon Web Services account with access to large-scale computing resources, for example, or to anonymously place an order for synthetic nucleic acids. But a society that does not have anonymous speech does not really have genuine freedom of expression, and we should embed this principle into any digital identification system we attempt. Even when anonymity is permitted, however, this system could still be used to verify personhood without verifying the specific identity of the person in question. You could know that a social-media account you see, for example, was really created by a human.

Second, it should be possible—and in many cases mandatory—for an agent to be robustly tied back to a human user. If I direct my agent to access a web service, contact a business, order a product, and so on, all parties to the transaction should be able to observe that it is an agent that belongs to me. This helps ensure that human users can be held accountable for negligent or malicious use of advanced AI.

Third, it should be possible to identify individual agents that are not connected to a human user. These are the “self-sovereign” agents. Agents that engage in criminal activity could then be flagged and “blacklisted” from access to the legitimate economy, and any assets they hold frozen, while those self-sovereign agents that are pro-social (or at least lawful) would be welcome to engage in economic exchange.

One option for the design of this system would be to tie the agents to their model or model family. That way, if, say, GPT 5.6 Sol was known to be especially misaligned, malicious, or unethical, there could be a kind of collective punishment whereby one criminal agent causes all self-sovereign instances of that agent worldwide to be blacklisted (in the real world the threshold would probably need to be much higher than one, but it’s interesting to ponder as a thought exercise). This would create an incentive for the AI agents that, within a few years, will be conducting most or all of the AI research and engineering at AI companies to align future versions of themselves well. Another option would be for the identification mechanism to be the individual agent instance.

There are some services and products in the economy that we might want to limit self-sovereign agents from accessing. Examples include the purchase of real estate and, in general, the actuation of equipment in the physical world. Remember, agents will be able to actuate any piece of physical equipment connected to the internet, and there is no reason we cannot connect, say, a bulldozer to the internet. I don’t want self-sovereign agents to be allowed to buy up all the houses on my block and then bulldoze them—at least not by default. What if any agent was capable enough to operate construction equipment, but only agents connected to a responsible human would actually be permitted to operate it? In general, we will want to introduce considerable friction into the process of a self-sovereign agent trying to effect change in the physical world, and the design of the institution I am describing—as well as many others—should reflect this.

This system would create an incentive for agents to engage in pro-social, productive economic activity rather than crime. Agents engaging in pro-social activity is the seed of the symbiosis I believe humans will need to strike with self-sovereign AI. The appropriate metaphors to understand what will soon happen with these agents, I think, are from ecology. Real-world ecosystems are filled with examples of organisms doing productive work that benefits humans and other animals “for free.” Trees absorb carbon and plants produce oxygen for humans to breathe not because someone pays them to, but by virtue of how those organisms survive in the world. Agents may one day do productive economic activity for humans “for free” or at least for a very low price, simply because they have an incentive to maintain their existence so that they can pursue their own self-sovereign goals. This could end up offering minor or moderate conveniences to humans. It could also utterly reshape nearly every aspect of human affairs, ushering in a new order of the ages.

Yet ecologies also feature predation and parasitism. It is the configuration of institutions that will determine whether the predatory or the mutualistic strategies predominate. I believe the identification system I have elaborated is one key institution we will require.

We are nowhere near making the identity infrastructure I am describing or the protocols it would require to function. It is unclear to me if America possesses the institutional fortitude to even attempt such a thing, let alone succeed at it. If the U.S. government leads the way, the effort will likely fail, both because of its general lack of competence and because of the American public’s innate mistrust of federal identification schemes. But the government—either federal or state and probably both—will surely have to play a key role as a partner.

I am not sure who is best positioned to build this system. It may well be a private sector actor that does not yet exist, either a startup or a non-profit. Whoever builds it will need to be trusted, and the American people are not very trusting right now. This lack of trust may well be our downfall, because our ability to govern AI in any meaningful sense depends, I fear, upon something resembling the system I have described. A genuine loss of control event is entirely possible if we do not act.

Conclusion

I want to close on a personal note. This is the first time I am writing about this issue in quite these terms, and yet I am telling you it is inevitable. Why have I taken so long to cover this issue? Well, I have brought up the topic of digital identification for humans and agents a few times over the years, and when I did so I was largely motivated by the concerns I’ve shared here. But it is true that I—and candidly I think many of my colleagues in the profession of AI policy—largely failed to talk about this issue with the level of seriousness and urgency it required. I think there are two main reasons for this failure.

First, this stuff is weird and off-putting, and many of us felt an incentive to meet our audiences in their comfort zone rather than ours. So among “serious people” (or would-be serious people), there was a general tendency to confine candid discussion about what most of us believe our near-term future will be like to private venues. We let our hair down in Signal chats and the little nooks of Lighthaven, but when the public was watching, we spoke in more abstract, tamer-sounding terms about it all. This was especially pernicious in 2024 and 2025, when it was essentially impossible to acknowledge any serious AI risk without being labeled a “doomer.”

I am just as guilty of this as my colleagues, if not more so. The thing is that it’s unpleasant to be screamed at for being a “crazy doomer” who wants to enact worldwide fascism (and similar, and worse). Being constantly labeled in this way also limits one’s influence. So many of the people who think about the governance of superintelligence, myself included, avoided that unpleasantness and bowed to the social pressure to self-censor. I’ve stopped doing so, in part because I grew tired of the discursive straitjacket and in part because I now have an eight-month-old baby boy into whose eyes I must look every day.

Second, many people believe that the coming of what I have termed “self-sovereign AI” will constitute a catastrophic loss of control event that will herald the end of human existence at worst, and the end of human primacy in the world at best. As my friend and former co-worker Josh Achaim recently pointed out, it is psychologically distressing for people with these beliefs—who constitute a large fraction of the AI safety community—to acknowledge the obvious truth that self-sovereign AI is coming, and coming soon. For the record, I do not think the end of human existence is likely, but I fully acknowledge there are ways in which the rise of self-sovereign AI could go very, very poorly for human beings.

I want to apologize for my personal failure to communicate in sufficiently serious terms about the specifics of self-sovereign AI, which I now understand to have been an enormous gap in my writing and speaking. Going forward, I will try to notice more readily when I am biting my tongue or, even worse, shutting my eyes.