Introduction
A core problem about the thing we today call “artificial intelligence policy” is that different people mean radically different things by the phrase “artificial intelligence.” I mean the term “radically” here in its literal sense: there are some who believe that AI systems capable of end-to-end autonomy of all human cognitive labor are coming soon, and there are others who assert, either explicitly or implicitly, that this is impossible, or who otherwise refuse to take the idea of this technology being built seriously.
Both groups are basically bullish: the latter group believes in “really good LLMs,” and the former group believes in “AGI,” “transformative AI,” “powerful AI,” and various other terms used to refer to future AI systems.
This creates a policy planning problem. Suppose that today’s large language models, and even, for the most part, tomorrow’s LLMs, do not give you much concern from a regulatory perspective, or you believe whatever policy challenges they pose can be solved by challenging but ultimately achievable adaptations of existing legal frameworks and institutions. This is, basically, my view about the current frontier of LLMs.
But suppose you also believe that there could be future AI systems with qualitatively different capabilities and risks, even if they may involve LLMs or resemble them in some ways. These future systems would not just be “smarter,” they would also be designed to excel at cognitive tasks where current LLMs fall. They would be more human-like in their ability to learn new skills, interact with the world around us (even if just the digital world), and solve problems.
These future systems would turn us all into Gary Marcuses. Every sane observer, including those must bullish on AI today, would look at the systems of the future, reflect on the LLMs of 2025, and conclude, “of course those older systems weren’t really thinking, or if they were, barely at all.”
We do not know when, or if, those future systems will be built, and we do not know what the characteristics or ergonomics of such a technology would be. My own guess is that they will be built for the first time sometime between 2029 and 2035, but I have substantial uncertainty about this. I would not be that surprised if it is much sooner, and I would not be at all surprised if it is much later.
Suppose these future systems really did require a fundamental reimagining of almost everything about our civilization in a way that profoundly implicates the government and every other institution we have. This is, of course, a radically different policy posture from the world of “really good LLMs.” And suppose further that even speaking about the future systems—not just talking about the problems they may cause but literally referencing them in any intellectually serious way—resulted in you being branded as an outcast in much of polite society.
A challenging policy problem, indeed.
I’d like to accomplish three things today: (1) briefly describe the difference between “really good LLMs” and “the future systems,” (2) sketch out a bit of how I expect the economy, politics, and government to evolve with the technology over the next decade, and (3) conclude with some reflections about how to write policy today under such significant uncertainty.
From “Really Good LLMs” to Transformative AI
My intuition is that the difference between the current systems (no matter how much scale is devoted to them, and no matter how good they get) and the hypothetical future systems come from two interrelated but distinct deficiencies. The first is long-horizon agency, or the ability to act with intention and in rational response to the outside world over long periods of time.
The second is continual or “online” learning, which people sometimes also describe as “long-term memory.” This is the ability to learn from your experiences in a way that reliably changes your future actions and improves your likelihood of success at future skills. LLMs today achieve primitive “memory” by manually jotting down specific facts and inserting those facts into the model’s context window every time you prompt it. This is in stark contrast to how you learn, which is by developing a representation of your past experiences and abstracting lessons from those experiences, like the takeaways you had from a hard project or earlier job.
I don’t know that these two gaps are the only ones lying between current LLMs and the future systems I have in mind, but for purposes of this essay, I’m going to assume they are.
Where We Are Headed (Part 2.5)
At the beginning of this year, I predicted, in descending order of confidence, that, by the end of this year, frontier LLMs would rank by some measures among the very best human coders, begin to automate meaningful fractions of research and engineering within the frontier AI labs, and discover new knowledge in mathematics. I also said it was possible that frontier LLMs would begin to show signs of making novel scientific discoveries. Each of those things has happened in some way or another.
Right now, though, there is enough complexity about the results that naysayers can easily brush them off. This is how it always goes in technology. Over the next year, these same results will recur in less ambiguous ways. What is happening will become more explicit, though no harder to deny for those whose business model depends upon denying it. Regardless, by the end of 2026, many more people outside of the AI community will be convinced that something big is happening.
Partially this will be because of headline results, and partially this will be because of technology diffusion. More people will experience AI in their daily lives, and especially in their work. Labor market dislocation, already visible, will get worse, though I think limited to junior employment in specific professions. However, if we have a weak labor market overall, I expect that AI will be blamed by many for all job losses, even if it only is responsible for a relatively small portion.
It will be challenging to gauge the problem purely from the discourse, because whatever worsening there is will be catastrophized by those with the incentives to catastrophize, which include the media, politicians, and people whose jobs are safe but who perceive their vested economic interests to be threatened by AI (like large, corporate owners of intellectual property portfolios, referred to in Washington as “content creators” or, occasionally, “artists”).
The data centers will become a flashpoint in all this. Increasingly, anti-AI politics will be conveyed through data center NIMBYism, and I expect some states to introduce laws intended to curtail their development. You will hear more, not less, about data center water use, even as the largest data centers move to closed-loop water systems that massively reduce the lifelong water consumption of the facilities.
The politics of AI will become coarser and less connected from reality. Increasingly the “AI policy community” will be a small part of “AI policy.” A populist backlash will be in the works, and enterprising politicians on both sides of the aisle will seek to take advantage of it.
The speed of drug development will increase within a few years, and we will see headlines along the lines of “10 New Computationally Validated Drugs Discovered by One Company This Week,” probably toward the last quarter of the decade. But no American will feel those benefits, because the Food and Drug Administration’s approval backlog will be at record highs. A prominent, Silicon Valley-based pharmaceutical startup will threaten to move to a friendlier jurisdiction such as the United Arab Emirates, and they may in fact do it.
Eventually, I expect the FDA and other regulators to do something to break the logjam. It is likely to perceived as reckless by many, including virtually everyone in the opposite party of whomever holds the White House at the time it happens. What medicines you consume could take on a techno-political valence.
In the other sciences we will also make leaps and bounds. Major unsolved problems in chemistry, physics, and mathematics will all be overcome. We will make breakthroughs in fields like formal verification, pointing the way toward dramatically more secure software. Nuclear fission reactors will be brought online, increasingly ambitious space infrastructure will be erected (including on the Moon), and new materials will be discovered. Many of these immensely promising innovations will take years to diffuse, though.
By the end of 2026 or so, the cyberrisks will become real. The surface area is much too broad for me to make predictions as to what specifically will happen, but I expect by that time it will be unambiguous that AI systems are being used by malicious actors to hack the digital world in ways that normal Americans notice. Government will do very little about this, but politicians will talk about it often. Solutions will come from new and existing businesses offering cyberdefense as an enterprise and (maybe) consumer product.
The law enforcement of the internet will not be the government, because the government has no real sovereignty over the internet. The holder of sovereignty over the internet is the business enterprise, today companies like Apple, Google, Cloudflare, and increasingly, OpenAI and Anthropic. Other private entities will claim sovereignty of their own. The government will continue to pretend to have it, and the companies who actually have it will mostly continue to play along.
The fundamental tension will not be because the AI companies are inherently bad actors, but because they will increasingly challenge the implicit authority of the state and most other existing power structures. That tension between digital technology firms and the status quo has existed for decades. In that sense, AI politics will be nothing new—just the same conflict, heightened.
At some point during the next decade, the “really good LLMs” will cross one or both of the gaps I highlighted above. Once both gaps are crossed, and we truly do have “transformative AI,” you’ll live through a period not unlike what I have described, but more so: faster, more intense, and always more explicit. Quite possibly far faster, more intense, and more explicit. It is at this point that the levees will start to break.
If these transformative AI systems are capable enough to run autonomous or functionally autonomous corporations (say, with a small number of human overseers), and no legal barrier prevents this, default economic incentives will encourage this. These corporations will feel more alive and natural—not like alive like a person, but like a microbe; not natural like a mountain, but like the wind.
Entities of this kind will prefer to deal with entities like them—not human-hybrid corporations. They will also naturally outcompete human-hybrid firms in virtually all sectors where such competition is permitted. If they are not based in the United States, they will be based elsewhere.
I do not have an intuition for how governments will respond by the 2030s. It is too difficult to predict the evolution of our politics for me to make confident guesses. But I remain confident that the vast (vast) majority of statutes that are on the books will continue to be in 2030, that lawsuits will proceed no faster than they do today, and that the average politician will be broadly adversarial rather than friendly to the AI industry. If the governance of AI comes down to a fight between the industrial-era nation state and capital, it is capital, being better suited to its environment, that will win in the end.
Historians looking back on the period between 2025 and 2035 are likely to describe it as a renaissance. And it will be. That does not necessarily mean it will be an enjoyable experience for most of the people who live through it.
Policy Planning with Radical Uncertainty
I do not know if the scenario above will play out. It is my rough guess, but I only have sufficient confidence in some parts this model of the future to make public policy tradeoffs in preparation for it. Others are too speculative for near-term policy action, even if they imply plausibly serious consequences. I have different standards for the level of confidence I must have about a prediction to make different kinds of decisions. Will I allow the model I’ve sketched above to influence my personal decisions? Sometimes, sure.
But will I allow this belief about the future to affect the recommendations I make about public policy, which could plausibly affect the lives of millions, and untold trillions of economic activity? I have much higher standards here. I need to have meaningfully higher confidence in a prediction to recommend policy that would entail real costs.
This is also why, for example, I found myself opposed to California’s frontier AI safety bill SB 1047. I suspected, at the time, that models would soon pose novel biological and cybersecurity risks. But I did not have enough confidence in the timing or in the nature of the threat to support legislation.
I said, at the time, that models did not exhibit sufficient levels of “system 2 reasoning” (the ability to reflect, catch mistakes, backtrack, etc.) to make me sufficiently concerned about near-term catastrophic risks to justify the cost of a new regulation, however light touch it may have been. I publicly committed to changing my position if we did see such capabilities. When OpenAI introduced their first “reasoner,” o1, I immediately suspected it was the precise breakthrough I had described. After a few weeks of reflection and experimentation, I concluded that it was. My confidence in the catastrophic risk threat model improved, and, a year later, I find myself supportive of California’s SB 1047 sequel, SB 53.
Why am I so conservative about this? Because regulation is inherently path dependent. SB 1047 itself, for example, would have fixed training-compute-based regulatory thresholds into statute, which became outdated before the bill was even vetoed by Governor Gavin Newsom.
All public policy involves conjecture about the future, which in turn locks us into assumptions made by specific people at a specific time and place. Those assumptions often prove to be wrong in subtle ways that would have been hard to know ex ante. This will be doubly true in AI, so one should in fact be more cautious, not less, when it comes to this domain of public policy.
This is roughly how I plan to approach questions of “AGI policy” over the coming years—a careful balance of responsibility and humility. It is likely I will err at least some of the time, but I will try my best.
But this is not all we can do. A great many actions are genuinely useful given what we know about current LLMs and what we can reasonably guess about their likely trajectory, regardless of the shape and timeline of transformative AI. I conceived of the AI Action Plan as an attempt to articulate a comprehensive set of such policy decisions. Some critiqued the Action Plan as a “no regrets” document, refusing to make hard tradeoffs.
I understand such criticisms, but I do not concur. I think “no regrets” is exactly the posture we want to strike at this moment. In general, we should not plan policy based on the impacts we speculate about of technological breakthroughs that have not happened. Even if it were politically feasible to do so (it is not), it would be unwise: we do not know what we are dealing with, or what problems we are trying to solve. As we gain more experience with the technology, the aperture for prudent policy action widens. It has already widened considerably, for me at least, in the last two years.
This, I believe, is the challenge for AI policy: to lay down a foundation that we believe will suit us well iftransformative AI does indeed become a reality, without assuming it will become a reality, within whatever political constraints we face. In parallel, one should also try to imagine the kinds of institutions that may be possible in the world of transformative AI—and lay down the foundations for their eventual construction.
Conclusion
It is daunting to grasp the contours of the transformation I am stabbing at. It is difficult enough to do it for the future with “really good LLMs,” harder still for transformative AI. I understand that the possibility of transformative AI is alien, conceptually confounding, inconvenient, and easier to ignore. It invites us—requires us—to ask uncomfortable questions.
It would be easier to brush it off, either by denying it or rendering “the future systems” unlawful somehow. I empathize with this desire, I do not look down on it, and I do not regard it with the hostility that I once did. Yet I still disagree with it. The future does not unfold by show of hands, not even in a democracy. The decaying structures of high industrialism do not stand a chance in this conflict, which has been ongoing for decades and which they have been losing for the duration of that period.
Yet we must confront potential futures with open eyes: given the seriousness with which the frontier labs are pursuing transformative AI, it would be tragic, horrendously irresponsible, a devastating betrayal of our children and all future humans, if we did not seriously contemplate this future, no matter the reputational risks and no matter how intellectually and emotionally exhausting it all may be.
There is a reason the phrase is “feel the AGI.”