9 Comments
User's avatar
Nathan Lambert's avatar

Great post. I will quibble with the computer use and robotics example, though, as those capabilities came into the models after substantial data was curated in the ecosystem. That data is much more defined and easier to create, relative to the messiest, most influential knowledge work.

i do agree we should understand how to manage strong recursion, and regulate it, even though I disagree strongly with the claims that it is coming very soon (claims not by dean).

Brendan McGuire's avatar

Dean, this is one of the clearest pieces I've read on recursive self-improvement. Separating weak from strong recursion is the kind of calibration this moment needs. Fear doesn't help us discern, and panic writes bad law.

Pope Leo XIV's new encyclical, Magnifica Humanitas, gives us a lens that fits your argument. He warns of a future where we gain "having more" without "being more" (§94). He also warns that "a more moral AI is not enough if that morality is determined by a few" (§107). The danger isn't only runaway machines. It's power gathering in too few hands, whether those hands belong to companies or to the state, as you rightly warn.

The Church's answer is subsidiarity and solidarity. We need smart regulation shaped at every level of society: labs, governments, universities, faith communities, families. Yes, RSI is a real risk. But if we manage it together, and judge it by its impact on all of society, AI can serve the flourishing of the whole human family, not a narrow few.

That will require us to come together in ways we never have before. AI can help us do that. But we have to care about the means as much as the ends.

— Fr. Brendan McGuire

Eskimo1's avatar

Thank you for saying this. While it’s true that RSI could impose serious harms like loss of control or extinction, those pale in comparison to the horrors of OpenAI being regulated without the long and thoughtful input of Dean and his wise friends (who all work at OpenAI).

Dean W. Ball's avatar

I understand you’re being sarcastic, but this post is filled with motivating reasons and concrete proposals for regulating OpenAI and other frontier ai companies.

Eskimo1's avatar

Honestly man, you and your pals are threatening all our lives and making a fuckton of money doing it. So yeah, you’re not gonna get a lot of charity when you go all “let’s not panic, lest we overregulate”. You guys are steamrolling over society, noboby asked for or wants your shit and now you’re telling us to just calm down while you put our lives at risk.

David Carlton Adams's avatar

Thank you for doing this disciplined work in identifying the conflation between a historically contextualized, ordinary form of RSI and a version in which no human is at the wheel. Something I notice in myself is that I tend to think less carefully when scared or rushed. And when we're facing arguably unprecedented pressures on both the fear and urgency axes, it can be hard to slow down and do the kind of careful thinking that is often required to gain clarity. One of the first things to go is the precision born of groundedness and proper calibration.

Robin Reese's avatar

Dean, your weak/strong distinction is clarifying, and your proposal to measure the share of merged code receiving no substantive human review is exactly the right kind of metric. Anthropic’s S-1 reports that Claude wrote over 90% of its codebase contributions in September 2026. What we don’t know is how much received meaningful human review.

But I’d question your reliance on corporate duties as a brake on strong recursion. Anthropic’s filing describes “closing the loop” as a competitive advantage and warns of falling behind rivals who do so more effectively. That doesn’t establish strong recursion, but it does reveal a commercial incentive to increase AI autonomy. The business model may therefore be pushing toward the very boundary you’re trying to define and regulate.

And how strong a brake are those corporate duties? Anthropic and your own employer are both public benefit corporations, whose boards balance shareholder interests, affected parties, and public benefit under a deferential legal standard. Under Delaware law, only stockholders meeting statutory ownership thresholds can sue to enforce that balancing duty. The affected public cannot.

So human control raises a second question: accountable to whom? METR received substantial access, but Anthropic retained authority over what evidence became public. Commercial confidentiality is legitimate, but shouldn’t independent evaluators have enforceable access and a protected means of reporting significant findings to regulators?

You close by warning against power becoming concentrated in the state, a concern many of us share. Yet weak recursion, bounded by the capital its builders can marshal, can also concentrate power among the institutions with the most capital.

How would your proposed framework guard against both forms of concentration?

anon's avatar

Every day, you work on building something powerful and dangerous, that no one fully understands, and that you admit poses a "genuine risk" of uncontrollable recursive self-improvement, all without bothering to seek the public's consent. In my opinion, that makes you one of the most morally rotten and spiritually weak people to have ever lived. Rest assured that when the rest of America finally realizes what you and your co-conspirators are doing, they won't be as nice to you as all your paid-subscriber shills are.

"So the LORD scattered them from there over all the earth, and they stopped building the city." Gen. 11:8.

JAMES VANN's avatar

Dean, a very nice essay. I am sure you thought it through, but why choose the value laden terms ‘strong’ vs ‘weak’? Especially given that the competitive market problem will confound attempts at regulation and control. Wouldn’t something like ‘contained’ vs ‘uncontained’ be more appropriate?