Holly's basilisk

When the world wakes up to the unacceptable danger of AI development, what happens to those responsible? The Berkeley trials, perhaps.

The defendants at a Nuremberg trial via Wikimedia. This could be you AI co bros^

The scariest thing about AI development is that we might not see the disaster coming, like in AI 2027. Many are hopeful that the earliest disasters will be big enough to convince everyone of what needs to be done, but small enough for us to survive. This is commonly referred to as a “warning shot”, and I have a take on them, which is 1) that it’s ghastly to hope for people to die to make the job of raising the alarm easier and 2) that we cannot afford to wait for them.

But let’s imagine that we do have a “successful” warning shot that rallies the world to a real solution like Pause. The danger is clearly demonstrated— a 10% chance of extinction suddenly feels real— and the public appetite to allow further flirtation with mass death is gone.

What happens to those responsible?

Oh, what’s that you say, defendant? You weren’t breaking the law by making a superintelligence? Nullum crimen sine lege, no crime without a law? That was a defense at the Nuremberg trials, too. It was only after the Holocaust that we had a concept of “crimes against humanity” because until then there had been no precedent for punishing people for attempting to exterminate an entire race. No one has made a race of killer machines before, either, but don’t think that means you’re off the hook. If a warning shot rallies the world into stopping the unprecedented threat of AI harms, the world will also be rallied against those responsible in unprecedented ways. The Berkeley trials, perhaps.

The AI companies have all been writing their own prosecution from the beginning. Talking endlessly about how dangerous advanced AI will be while forging ahead anyway. Begging to be regulated in public and lobbying against regulation in private. Revising their safety commitments down and down.

This quote from an Apollo Research evaluation of Claude Opus 4.6 reads like something you’d find in a fallout shelter in a video game.

On February 28th, 2026, during the Iran War, Anthropic was involved with one of the worst war crimes against civilians since the Vietnam War, with 156 dead, including 120 children. That won’t be forgotten. After the warning shot, it’s going to look like an obvious pattern of reckless endangerment.

The Nuremberg Defense was that officers were “just following orders”. What I call the Anthropic Defense is that “the counterfactual made me do it”. Often it’s “but China”. Very often it’s “but the AI race” i.e. “The Incentives don’t reward us doing the right thing, so we can’t”. What it amounts to is that Anthropic can’t take any more responsibility than a tick above their worst competitor (but for this they are supposedly “the good guys”). It’s convincing enough to motivated listeners— who want to work at Anthropic, or work with them, or want their valuation to go up. Eichmann found it easy to “just follow orders” when he was architecting the Holocaust with the support of his German military superiors. But it’s going to sound pretty lame when Anthropic is accounting for real blood spilled in front of an audience of their victims.

I’ve noticed in my time crusading for Pause that there’s a sort of Pascal’s wager going on where people are afraid of being left out of the elite in a superintelligence future (maybe even an extreme scenario like Roko’s basilisk torturing them for not helping to build superintelligence faster), but they aren’t afraid of what would happen to them in a Pause future. They know Pause is the humane option that cares about human life, protecting them from suffering, and therefore easy to take for granted as a fallback option.

To those people I say, consider that the world isn’t just going to forgive you for endangering humanity. You’re only getting away with it now because the public can’t keep up with all the unprecedented risks you’re running. If there is a “warning shot”, you’re going to be tried and you’re probably going to jail. Call it Holly’s basilisk if you like.