Thank you to the organizers of The Curve for hosting me and giving me the opportunity to speak, and to the Rockefeller Foundation for facilitating the reflections below. Thanks also to Virginia Postrel for inspiring the title of this essay.
—
I spent the last few days wandering the ancient streets of Bellagio in northern Italy. Today this city exists mostly for tourism, but through history this has been a hard-nosed place of military strategy and commerce. Positioned at the tip of a peninsula on Lake Como, no water-bound trade could pass through this region without catching the watchful eye of they who occupied Bellagio.
There were the Celts, then the Romans. After the Roman Empire collapsed came the Lombardi and the Franks. Protective walls had to be erected at the city borders, since this newly fractured world had no great empire to ensure its safety.
More advanced industry gradually formed during the late medieval and early modern periods. In particular, the city became a center of silk production, whose traditional manufacturing process still lingers here today.
At the peak of the Bellagio promontory lies a site where, 2000 years ago, Pliny the Younger had a villa. During the golden age of Europe’s aristocracy, the site became the compound of Milanese aristocrats who christened it Villa Serbelloni. In the era of budding European capitalism, it became a hotel—though it remained in the hands of the Duchess of Serbelloni. And in 1959 that family passed the property onto the stewards of the fortune of a new emperor, the foundation of John David Rockefeller.
I have found this property to be a fitting place to ponder the transformation of institutions.
—
“Institution” is a funny word. I despise it, in fact. It is cold and dry, bereft of humanity. It is a word that sounds like it implicates concrete and steel, when really, institutions are composed of human beings.
Institutions are not “organizations.” When you think of an institution, you shouldn’t think of a building; you should think of people. The institution of Congress is not the Capitol Building, but instead all the people who work inside it, and their rules, habits, norms, rituals, preferences, ways of relating to one another, and so forth.
All institutions are technologically contingent, based as they are upon a vast complex of assumptions, almost wholly unstated, about what is possible and what is impossible, about what is hard and what is easy. And it is our technology that determines what is hard for us to do and what is easy. As technology makes new things possible, and eventually makes them easy, institutions must be transformed.
Think, for example, about the institution of science.
Picture yourself in a present-day scientific lab. Look at the equipment, all of it designed for individual humans to place individual samples for one-by-one analysis, in service of writing individual “papers” with frozen-in-time “results,” all funded by individual grants (usually from the government or a large philanthropy) for individual scientific micro-endeavors.
Imagine that you are a cook, and you just made a cake in your kitchen. You’ve made a delicious cake, and you’d like to start a business making 1,000 of them a day. So you replicate your kitchen 1,000 times over—you buy 1,000 residential ovens, 1,000 standard mixing bowls, 1,000 bags of flour. And you hire 1,000 humans to follow your recipe, each making their own cake in the various kitchens you’ve built.
Of course no one would do this. And yet this is not that far off from how we today “scale science,” and in some ways we are even less efficient.
What you should do instead, obviously, is build a factory with the ability to make 1,000 cakes at the same time. This was, at one point, a new type of institution that entailed distinct organizational structures (the modern corporation, for example), new relationships of workers to firm owners, novel patterns of work, and much else. The factory enabled and necessitated new technology: in our example, industrial ovens, wholesale purchase of ingredients, and the like, in quantities that would be alien in a residential kitchen. Similarly, it required new occupations that do not map cleanly to their pre-industrial analogs (consider the “chef” or “baker,” for example, versus the “batter-vat cleaner”).
Standing in a pre-industrial residential kitchen, it would be difficult to imagine a factory, partially because there are numerous complementary innovations a factory requires (for example, the ability to manufacture and power industrial-scale cooking equipment), and partially because imagination is hard. Unrealized ideas are some of the most fragile things human beings produce. Standing in a present-day scientific lab, it would be similarly difficult to imagine the industrial-scale science of the future, and all of the complementary technological and institution innovations it will require.
But we can try. Imagine a new industrial science automated with robotics in the world of atoms and agentic frontier models in the world of bits. Consider millions of experiments in parallel, generating data and analysis in a continuous stream, but only incidentally for a human audience. A ceaseless machine interrogating and manipulating nature with greater finesse by the hour, at once magnificent and terrifying, as all great machines are.
In this new world, the human-relevant scientific unit of account is no longer the “paper” or the “dataset” or even the “experiment.” Instead the thing humans care about becomes the creative question, the daring moonshot, the industrial objective. How do we create an incentive for future practitioners of scientists to ask great questions? Do we do so today, or do we mostly incentivize them to write great grant proposals?
—
When I and many others say that “AI will challenge our institutions,” we are burying the lede beneath frigid nomenclature. What we are saying is that AI upends the way most people who do most economically valuable things conceive of their work, their organizations, and, ultimately, themselves.
We face a question: do we try to reform, improve, adapt, refine, or update our institutions? Or do we start from scratch, building new things altogether? And can we—America, the West, humanity at large—stomach such momentous change?
It is a question as old as technology. But both institutional reform and creation get harder over time, because as our civilizations grow older, and as we become wealthier (due to prior, successful institutional innovation and co-evolution with technology), change gets more difficult. Our bones stiffen. We have more to lose. We become tired.
Like most things, I suspect our path forward will require both reform of existing institutions and the forging of new ones. In some cases the new will accumulate in sedimentary fashion over the old. In others, the new institutions will outcompete the existing ones, sometimes viciously. A great many storied institutions will be—pardon—railroaded by the technological wave that is building. Based on what I have argued here about what institutions really are, I hope you understand that this will be hard, emotionally and otherwise, to internalize.
A great many people, once they realize what is underway, will understandably fight the new institutions. They will seek to entrench the status quo, to reject entirely the possibility of change so radical. And they will ask you which side you are on. You will face a tough choice.
Anytime I walk around a scientific lab, I feel the anxiety and frustration Henry Ford must have felt when he studied the pre-assembly-line factories. Everywhere I look, I see new empires and emperors dying to be born. And everywhere I look, I see the institutions of old ready to fight tooth and nail.
—
A few days ago I was in the San Francisco Bay Area, the birthplace of the coming revolution in our institutions. My primary business was attending an excellent conference called The Curve, which brings together an eclectic mix of delegates from the institutions of old, and the institutions which today are struggling to be born.
Several attendees remarked upon the clash of Washington politicos and San Francisco technologists. I noticed this too, and thought of cowboys and Indians sizing each other up, readying for battle, yet dwarfed by the size of the terrain and the scale of the ideas over which they feud. I tried, and ultimately failed, to determine which side I was on.
I came to The Curve with an offer. Not so much with a proposal—though I did have one—but with an outstretched hand. Fundamentally, I am aware that institutional transformation—especially so much of it all at the same time—is going to be resisted by many, if not most. In a fraught world replete with risks, what worries me above all else is that our society’s efforts to fight change will encase the institutions of the present in amber.
These fights will have very little, perhaps nothing, to do with matters of frontier AI safety. Some of them will be fights worth having; many will be vicious attempts to quash fragile visions of a better future. Shockingly few people are on the side of techno-optimism, of radical technological change and attendant institutional transformation. The future has very few friends.
But The Curve was filled with friends of the future. Some are accelerationists, relatively less concerned about AI risks—though few would deny those risks are non-existent. Some are AI safetyists, relatively more concerned about AI risks—though few would support regulating AI such that that its positive uses are rendered impossible. Some of us try, and ultimately fail, to determine which side we are on.
These friends of the future are divided. They believe they are in a rivalrous competition with one another, an arm wrestle on the verge of becoming a fistfight. I believe, and always have believed, that this is wrong. The friends of the future should be allies, not enemies.
I believe the logical starting point for such an alliance is the federal preemption of problematic state AI regulations. I put forth a proposal to advance this discussion, but I hope others propose their own. Anton Leicht has written thoughtfully about what the political contours of such a compromise could look like. I have little to add to his analysis.
There may come a day when we are forced to make tradeoffs that do cause the accelerationists and the safetyists to become rivals. Let that day come when it must. In the meantime, I submit to you that the few friends the future has should work together.
—
The future I hope for will be hard-won. It will require more than just reasonable compromises on AI legislation—much more. It will require more than merely pushing back on ill-conceived regulations or cutting red tape. These things are small slivers—shadows, really—of what I have in mind.
Instead, the future I hope for will have to be advocated for with the utmost zeal and fierce belief. It will demand cajoling, coaxing, persuading, and not a small amount of fighting. Most importantly, though, it will require imagination: a willingness to invent the assembly line and the factory, to cast aside the old and start fresh.
Walking through the ossified streets of a northern Italian commercial-hub-turned-museum, I wonder and worry about whether the contemporary West can muster this zeal. If we can, it will probably be because of the combined efforts of a small group of people who saw the future early and decided to befriend it, a ragtag band of marchers through the institutions.
As you consider whether you want to join the march, look around you. Have the institutions of the present day served you well? Do they seem healthy? Do they seem repairable? Are you happy about the status quo?
I know my answers to these questions. My march, therefore, will proceed. Through mud and sand, in freezing rain, against bitter winds, under the glaring sun, I will march with the friends of the future, step by step.
This post updated me somewhat towards thinking that an arrangement might actually make sense.
*However*, the proposal you've floated doesn't seem anywhere close to balanced.
As Anton points out, what you've proposed essentially just offers the AIS movement SB53 at a federal level. That's not really worth much, the frontier labs aren't going to give up on California any time soon.
In exchange, accelerationists would recieve protection from an oncoming Tsunami of state legislation *and* forgiveness for years of unhinged and bad faith attacks on the AIS movement (which seem likely to resume even if there was some kind of arrangement).
A more balanced deal would likely include much more substantial concessions, such as a properly funded US AI Security Institute (on the level of the UK one). Actually, it doesn't really make much sense to think of this as a substantial concession given a) how acceleratory AIS work has been overall (unfortunately) b) that it is in everyone's interest to have a better understanding of model risk c) that safe models are actually more useful models.
I think change is upon us, whether we befriend it or not. I'm concerned I might not be up for the challenge, but I'm joining you on the march because there's only one way to find out.