If a young person asks me "but how to do this in policy", I have but one piece of advice: go into the weeds. Talk and argue through something that is clear. Don't ask for more money for transit, or for $20 million for transit, or for "better spacing". In high school, a friend of mine actually observed bus routes and wrote up recommendations for the MBTA about which specific stops should be removed. That is where I advise starting.
1. I broadly agree that there is much work to do. There is no partial credit in preventing extinction.
2. I'm not sure if you disagree, but I consider the recent letter to be a small step in the right direction due to it's role in building the salience of this topic and the salience of the main policy goal: a moratorium on building ASI. Just a couple years ago basically nobody said the word "superintelligence" outside niche intellectual communities; we've come a long way and there is much more to do; we shouldn't overly praise small steps, but if they are steps in the right direction (personally I think there are many "right" directions), that's something.
3.
> Ideas like “ban superintelligence” are policy proposals based on vague sentiment, just like “defund the police” was a policy proposal based on vague sentiment. The risks for social harm are similarly large.
I think it's worth noting that there are a bunch of other vague statements like this across the political spectrum. For example:
In fact it seems like every political rallying cry is under-specified with respect to its policy recommendations. It still seems potentially useful to have such rallying cries. I'm not sure how to allocate credit between activists and congressional staffers, but both seem important.
4. I get the sense that your cruxes in this matter are specific details such as, how to define ASI and when do we implement a ban, what does the ban look like such that we can assess how costly would a ban be—especially regarding other large-scale risks, and who would decide and how when to lift a ban. Is that right?
Many people aren't thinking about this topic at all, let alone what the broad shape of the policy solution should be, let along the specifics. The statement in the letter is far less obvious to most people than "murder is bad". There is substance here, just maybe not substance on the parts that you find cruxy. Other high-level solutions that other people sometimes advocate for include: treat AI like normal technology and let the market+liability solve the problems, do a Manhattan Project racing to ASI and use ASI to force China into submission. The letter is saying a thing that is in opposition to these plans—it provides some bits.
Anyway I've spent too much time writing this comment while there is, I agree, lots of other work to be done.
I too genuinely marvel at planes and airpods, but this seems a bit questionable.
The superintelligence ban looks unambiguously like a work of politics, not of policy. Policy work doesn’t involve appealing to Meghan Markle’s authority. This is a political slogan thinly veiled as a policy proposal: defund the police, freeze the rent, build the wall. The point of these slogans isn’t “here’s what we should do about this” so much as “we need to do something about this.”
On this view, criticism of political rhetoric based on its policy merits is itself more sophisticated political rhetoric.
I concur. As an older Gen Z coming from a remote place of India, I can’t help myself from appreciating the wonders of modern medicine and technology at large. This degrowth mindset towards tech that we have nurtured in the last decade is part of the culture wars—but even the strongest critique can’t deny that we live like monarchs only because technology has subsidised our lives.
Beautifully written. The petition is pure vibes with no definitions, no thresholds, and no policy behind it. I liked the craftsmanship metaphor, though it took a while to reach the point.
If a young person asks me "but how to do this in policy", I have but one piece of advice: go into the weeds. Talk and argue through something that is clear. Don't ask for more money for transit, or for $20 million for transit, or for "better spacing". In high school, a friend of mine actually observed bus routes and wrote up recommendations for the MBTA about which specific stops should be removed. That is where I advise starting.
My easier life would be becoming a full time analyst, the one that matters more is building models today.
1. I broadly agree that there is much work to do. There is no partial credit in preventing extinction.
2. I'm not sure if you disagree, but I consider the recent letter to be a small step in the right direction due to it's role in building the salience of this topic and the salience of the main policy goal: a moratorium on building ASI. Just a couple years ago basically nobody said the word "superintelligence" outside niche intellectual communities; we've come a long way and there is much more to do; we shouldn't overly praise small steps, but if they are steps in the right direction (personally I think there are many "right" directions), that's something.
3.
> Ideas like “ban superintelligence” are policy proposals based on vague sentiment, just like “defund the police” was a policy proposal based on vague sentiment. The risks for social harm are similarly large.
I think it's worth noting that there are a bunch of other vague statements like this across the political spectrum. For example:
We stand with Israel (https://www.congress.gov/bill/118th-congress/house-resolution/771).
We should reduce the deficit.
Build the wall.
Immigrants are welcome here.
In fact it seems like every political rallying cry is under-specified with respect to its policy recommendations. It still seems potentially useful to have such rallying cries. I'm not sure how to allocate credit between activists and congressional staffers, but both seem important.
4. I get the sense that your cruxes in this matter are specific details such as, how to define ASI and when do we implement a ban, what does the ban look like such that we can assess how costly would a ban be—especially regarding other large-scale risks, and who would decide and how when to lift a ban. Is that right?
Many people aren't thinking about this topic at all, let alone what the broad shape of the policy solution should be, let along the specifics. The statement in the letter is far less obvious to most people than "murder is bad". There is substance here, just maybe not substance on the parts that you find cruxy. Other high-level solutions that other people sometimes advocate for include: treat AI like normal technology and let the market+liability solve the problems, do a Manhattan Project racing to ASI and use ASI to force China into submission. The letter is saying a thing that is in opposition to these plans—it provides some bits.
Anyway I've spent too much time writing this comment while there is, I agree, lots of other work to be done.
Check this out: https://foxchapelresearch.substack.com/p/i-think-substrate-is-fraudulent-part
I too genuinely marvel at planes and airpods, but this seems a bit questionable.
The superintelligence ban looks unambiguously like a work of politics, not of policy. Policy work doesn’t involve appealing to Meghan Markle’s authority. This is a political slogan thinly veiled as a policy proposal: defund the police, freeze the rent, build the wall. The point of these slogans isn’t “here’s what we should do about this” so much as “we need to do something about this.”
On this view, criticism of political rhetoric based on its policy merits is itself more sophisticated political rhetoric.
I concur. As an older Gen Z coming from a remote place of India, I can’t help myself from appreciating the wonders of modern medicine and technology at large. This degrowth mindset towards tech that we have nurtured in the last decade is part of the culture wars—but even the strongest critique can’t deny that we live like monarchs only because technology has subsidised our lives.
Beautifully written. The petition is pure vibes with no definitions, no thresholds, and no policy behind it. I liked the craftsmanship metaphor, though it took a while to reach the point.
dimmite praeterita; specta ad futuris.
Well said!