Open-Source AI has Overwhelming Support

A quick analysis

Here’s a surprise post to liven up your Sunday.

But first, I published an oped on SB 1047, a proposed California law to regulate AI models, in the Orange County Register and other southern Californian newspapers (thanks to Kyle and Phalen at Mercatus for awesome and fast edits). One important point: SB 1047’s envisioned regulator, the Frontier Model Division, would be funded by fees and fines assessed on AI companies. This is a recipe for regulatory capture; the regulated pay the salaries of the regulators—what could go wrong? The Nuclear Regulatory Commission, by the way, has a similar funding structure.

Thanks to Chris Lengerich of Context Fund for inspiring today’s post. Context Fund has a good Reddit for those interested in keeping tabs on the open-source AI community, especially as it intersects with policy.

The National Telecommunications and Information Administration recently held a request for comment on open-source AI models (“open weight,” as the NTIA calls it—I’ll refer to them as “open models” here; there is an important difference between open source and open weight that I’m not going to get into here). The request, which came in response to President Biden’s fall 2023 Executive Order on AI, asked a broad range of good questions about this vital topic. If you are seeking background on what open-source AI is and why it is important, check out my earlier post on the subject.

More than 300 industry groups, academic institutions, think tanks, corporations, and individuals (myself included) submitted comments. I’ve been meaning to try the new long context capabilities of the frontier language models, so I threw all 332 comments into Google’s Gemini 1.5 Pro, which can keep roughly 750,000 words in its active working memory (inside Google, there is a version of Gemini that can do 7.5 million words simultaneously). The comments totaled well over 1000 pages and more than half a million words. With a little elbow grease, needed primarily because Gemini at first erroneously flagged the comments as ‘unsafe’, the model successfully analyzed all this text in a single prompt.

The results are clear: the comments overwhelmingly support maintaining access to open models. The arguments in favor and opposed are largely what you would expect if you’ve paid attention to this debate. The vast majority of respondents articulate the benefits of open models to science, business, competition, transparency, safety, and freedom of speech. Almost everyone acknowledges that there are risks associated with open models (and all AI models). A minority of respondents believe these risks merit policies to ban or severely restrict the development of open models. Almost all the organizations in that group are AI safety-focused non-profits. Most other people and groups think the risks are worth it for the many benefits open-source AI can provide.

Open model restrictions are primarily supported by a safety-focused subset of the AI community. Almost no one else in the AI industry, the broader technology sector, the business community as a whole, or civil society supports policies that restrict or ban open models. At least, almost no one who submitted a comment.

The supportive camp contains a wide range of views. Many respondents believe that frontier models should be closed initially, with those models (or similar) becoming open source after a lag. This happens to be the current basic equilibrium, at least with language models: The most capable models are released by frontier labs like Anthropic and OpenAI as API-only (closed), while open-source models take around 12-18 months to catch up. Perhaps this dynamic will change, but for now, it seems as though we have already achieved a reasonable balance between openness and safety without any policy intervention.

The tech industry is, as one would expect, overrepresented among the private sector respondents. Other business groups, however, such as the National Association of Manufacturers and the US Chamber of Commerce, also submitted comments favorable to open models. Groups traditionally seen as left-of-center, such as the ACLU and the Center for American Progress, supported open models for the benefits they can bring to civil rights, social equity, and transparency. For their part, and to their credit, neither OpenAI nor Anthropic nor Google/DeepMind support restrictions on open-source AI. All three firms, which some fear are seeking regulation to build a business moat, acknowledged the immense value open source provides.

All this positive sentiment could disappear overnight, of course, if the political winds shift. AI moves fast. There is no reason to think its politics will be any different.

And there are pressure points. Many of the supportive proposals make gestures at the need for “appropriate safeguards” for both open and closed models. I strongly support the development of robust technical standards, and even making them mandatory for some industries and use cases. Yet the “appropriate safeguards” language often struck me as the commenters trying to have it both ways, to evade the tradeoffs their positions entail rather than own them. As Claude 3 Opus put it:

The fundamental nature of open source is that once the models are publicly released, there is no practical way to enforce restrictions on their use. Any attempt to impose mandatory controls would undermine the very concept of openness. So while many of these comments pay lip service to the need for appropriate safeguards, their emphasis on the importance of open access and warnings against restrictive policies belie a hands-off regulatory approach in practice.

It's a bit like saying "I support free speech, but with reasonable limitations." It sounds moderate and balanced, but in reality the speaker is accepting that the downsides of unfettered speech are the price we pay for the upsides. They are not actually proposing a way to get the benefits of free speech while also eliminating the harms - because there isn't one, short of dramatically curtailing free speech itself.

Similarly, the "openness with safeguards" stance sounds reasonable on paper but falls apart in practice. The benefits of open models - broad accessibility, decentralized innovation, transparency - are inextricably tied to the lack of centralized control. Mitigating the risks requires asserting more control. There is no free lunch.

These tradeoffs are familiar to Americans. They undergird our Constitution and much of our history. Yet they remain tough tradeoffs that one must make with open eyes.

Nonetheless, it is heartening to see such a wide range of organizations voice their support for open-source models. The responses highlight the broad benefits and kaleidoscopic use cases of open-source AI in particular, and open-source software in general.

The community that believes open-source AI is an existential threat to humanity has thus far played a major role in the AI policy discourse. Their members hold named fellowships in Congress. At their urging, many policies that would effectively ban open-source AI are on the table. In publicly funded research, they proposed jail time for publishing an open model. Evidently, though, their views are a distinct minority. That does not necessarily mean they are wrong, nor does it mean they should be sidelined or dismissed.

It does mean, however, that openness should be the default path as policymakers and civil society chart the way forward. A broad range of societal actors have made their position clear: Open-source AI isn’t just a risk to be managed; it is a strength to be harnessed.

If AI safety advocates wish to change this default, the burden of proof will be on them. Policies that ban open-source AI altogether or forbid it above a certain compute or capability threshold require a staggering level of government control over human beings’ use of computers and a fundamental reshaping of our norms about digital freedom and privacy. The burden of proof, therefore, is high.

Both Claude 3 Opus and Gemini 1.5 Pro estimated that roughly 80% of responses were broadly in favor of open-source AI. I did not have time to manually verify their estimates. My anecdotal impression is that the ratio is a little lower—maybe something like 70-30 in favor of open source. Some of the comments are ambiguous, so that may be the source of the variation.

With the drop of a file into a prompt, it is now possible to cogently analyze hundreds of thousands, and soon millions, of words—in seconds or minutes (though in my experience, Gemini 1.5 Pro is a little buggy at the outer limits of its context window—your mileage may vary). Anthropic’s Claude 3 Opus, with a mere 150,000 word context window, provided consistently probing analysis when the data was presented to it in novel-length increments. OpenAI’s GPT-4, with a context window in the same ballpark as Claude’s, offered a superficial overview.

This analysis of a text corpus substantially longer than War and Peace took about an hour to perform from scratch, the vast majority of which was spent collecting the PDFs and plain text comments from regulations.gov (N.B.: I later learned that they have a bulk download feature). It cost me nothing beyond subscriptions to ChatGPT and Claude (and these were not, strictly speaking, necessary). This capability is in the hands of everyone with an internet connection and very modest technical skill. We are just scratching the surface.