Here's what we’ve been working on for over a year: The first US government-commissioned assessment of catastrophic national security risks from AI — including systems on the path to AGI. TLDR: Things are worse than we thought. And nobody’s in control. x.com/billyperrigo/s…
Here's what we’ve been working on for over a year: The first US government-commissioned assessment of catastrophic national security risks from AI — including systems on the path to AGI. TLDR: Things are worse than we thought. And nobody’s in control. x.com/billyperrigo/s…
1/ We started this work with concerns, but no preconceptions. We knew there were solid technical reasons that AI could eventually pose catastrophic risks. But we went in looking for reasons to change our minds. We found the opposite.
2/ Our overriding goal was to get to the truth. To do that, we had to do more than just speak to policy and leadership at the AI labs. We also connected with individual technical researchers, many of whom are way more concerned than their labs let on in public.
3/ Many of these folks came forward on condition of anonymity to share stories. Let me tell you some of the most insane stuff we learned.
4/ First off, inside one lab there’s apparently a running joke that their security is so bad that they’re doing more to accelerate the AI capabilities of US adversaries, than the adversaries themselves are. Truly crazy. But this is where we’re at.
5/ In December we quietly polled a handful of frontier AI researchers and asked them: What’s the chance we end up on a path to a catastrophic AI outcome, *during the year 2024?* We expected <1%. But no: Lowest we got was 4%. Highest: up to 20%. That's a wake-up call.
6/ One researcher said he was concerned that if $MODEL was ever open-sourced, that would be “horribly bad”, because the model was so potentially good at persuasion that it could “break democracy” if it was weaponized. Still expects it to happen, within 18-36 months.
7/ Another frontier AI researcher says: “It’s insane that we’re scaling without having a good way of telling if a model can do dangerous things or not.” Sure seems like it.
8/ Does this mean the AI labs *are* insane? No. In fact many of them *want* to do the right thing. But it’s like I said: nobody’s in control. Here’s what I mean:
9/ We visit one frontier AI lab. An executive there tells us, “we really wish $COMPETITOR wouldn’t race so hard.” A few weeks later, we speak to $COMPETITOR. And they tell us the same thing about the first lab.