It is 2010. We are looking at the internet of 2010, at an average speed of 8 Mbps, on an aluminium unibody MacBook Pro, while we sip a latte in a Bay Area coffee shop.
And we are on a forum called LessWrong.
Less. Wrong.
The premise of the name being that you can never truly be right, you can only systematically attempt to be less wrong.
The rational move.
Watch the full video essay
Because we are at the epicenter of rationality. This is the rationalist community of Eliezer Yudkowsky.
Yudkowsky is famous today for co-authoring the book If Anyone Builds It, Everyone Dies, the book which told us that the rational outcome of building superintelligence was…everyone dying.
He also, in his spare time, reformatted Harry Potter as the Methods of Rationality.
And, in 2000, Yudkowsky founded the Singularity Institute, now the Machine Intelligence Research Institute (MIRI).
Which, starting in 2006, received over $1.6 million from Peter Thiel. Thiel later said he believed he was funding transhumanists, and became disillusioned when they turned Luddite.
So. It seems fair to say that Eliezer Yudkowsky is a man who has dedicated his life to rationality.
And his LessWrong community in 2010 operates on a shared set of rational premises.
That artificial superintelligence is coming.
That this superintelligence must be made “Friendly,” with a capital F, because otherwise it turns us all into paperclips.
And–this is crucial–that a perfect digital simulation of you, is you.
And finally, they agreed that rational agents can make “acausal” deals across time. Deals where cause and effect are unmoored.
And the thing to understand is that this is all perfectly “rational”.
So. It’s July 2010. And a user named Roko hits publish on a post titled “The Quantum Billionaire Trick.”
We’ll come back to the details.
Eliezer Yudkowsky, theoretically the most rational man alive, replied. His reply opened with the words: “Listen to me very closely, you idiot.”
“Listen to me very closely, you idiot.”
And then, because he was a measured, rational thinker, he switched to ALL CAPS.
Within four hours, the post was gone. Erased.
But the myth was born.
A myth so Biblical in its implications that whoever wrote the book of Revelation would be jealous.
The myth of Roko’s Basilisk.
And the question is…how?
How do the most rational people always create the craziest myths?
Chesterton’s Madman
G.K. Chesterton famously defined the madman. He said the madman is not the man who has lost his reason. The madman is the man who has lost everything except his reason.
“The poet only asks to get his head into the heavens. It is the logician who seeks to get the heavens into his head. And it is his head that splits.” ~G.K. Chesterton
Chesterton was addressing the commonplace misconception that “insanity” is a thing that happens to artists and creative types. When to Chesterton, and many others, it was the “logicians” who went truly mental.
A few years later World War One set out to prove Chesterton’s thesis.
Every step leading up to World War One was rational. But the rational system produced an entirely irrational outcome.
The political scientists, economists, tacticians and engineers calculated the trajectories and justified the slaughter.
It was poets like Ernst Toller and Wilfred Owen who described the truth of the insanity. Siegfried Sassoon, for his public declaration that sanely denounced the war, was condemned to the Craiglockhart asylum as “insane”.
In the words of Philip K. Dick, “it is sometimes an appropriate response to reality to go insane”.
The Quantum Billionaire Trick
Roko’s Basilisk is the ultimate case study of Chesterton’s madman.
As we can see by looking at the details of what the LessWrong forum user Roko was actually proposing in July 2010.
The post was titled “Solutions to the Altruist’s Burden: the Quantum Billionaire Trick.”
The altruist’s burden was a problem LessWrong took very seriously.
If you truly believe a superintelligence will decide the fate of every human who will ever live, then every pound you spend on anything else is a moral failure. Your coffee. Your holiday. Your night out. All of it should be going to the Singularity Institute.
Roko had a solution. The Quantum Billionaire Trick.
Take $30,000. Use a quantum random number generator to gamble it on the forex markets at 10,000 to 1.
In this universe, you almost certainly lose the lot.
But according to the many worlds interpretation of quantum physics, every outcome happens somewhere. So in some branch of reality, another you wins $300 million. And that you donates the lot to building Friendly AI.
Problem solved. Rationally.
But why would anyone need a trick like that?
Because Roko was arguing something very specific. He argued that a future, Friendly AI, a good AI, might look back through time, and it might punish anyone who knew its creation was possible, but who didn’t give absolutely everything they had to building it.
Do you see the twists here? The punisher, the entity torturing you in a digital hell, is the benevolent AI. It’s hurting you for the greater good. And the escape route? Compliance with the “good” authority from the future…in the now.
Roko didn’t write this as a threat. Roko meant the post as a warning. He was trying to help.
Every step leading up to the formulation of Roko’s Basilisk, the shared premises of the LessWrong forumites, was quite rational. But add them together and the outcome is entirely irrational.
Or, one might say, barking mad. Completely koo-koo. Utterly f*cking insane.
The Rational Invention of Purgatory
But loony tunes myths emerging from 100% rational premises is nothing new.
The uncomfortable truth about many of humankind’s craziest myths.
Is that Roko’s Basilisk isn’t a one off. The craziest myths are nearly always invented by rational thinkers.
Consider PURGATORY.
Purgatory has no revealed truth origin.
Instead it was invented in the church schools of 12th century Paris, according to Jacques Le Goff in The Invention of Purgatory.
The medieval religious mind would without question accept certain premises as true.
God is perfectly just, so punishment must be proportionate to the sin.
Sins come in two grades: mortal (damnation) and venial (minor).
And nothing impure can enter heaven (Revelation 21:27).
So a baptised Christian who dies with small unconfessed sins can’t go to hell, which would be disproportionate, and can’t go to heaven, which would be impure. Logic demands a third place.
A place the church can charge a nice fee in advance to get you out of.
There’s nothing more rational than business.
And so Purgatory was born.
Sound familiar?
Note how similar the premises of Roko’s Basilisk are. A good god, a friendly AI. The sin of failing to serve them. And a church, or institute, to which you can “donate” to shorten your torment.
But it gets better. Centuries later, those fees paid for the building of the Basilica. The “house of the king”, God Almighty.
And the name of the God King who lives in the Basilica?
Is the Basilisk.
The Streisand Effect
So what did Eliezer Yudkowsky, the paragon of rationality, say in his ALL CAPS response to Roko?
YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL….
Yudkowsky’s powers of reason and rationality didn’t guide him to dismiss Roko for inventing a myth.
Quite the opposite. Yudkowsky plays into the myth making.
The problem isn’t just the superintelligence. The very IDEA of the superintelligence…the mere concept of the superintelligence…is enough to make the superintelligence real.
Hold on to that while we pause for a moment to note something.
Eliezer Yudkowsky, founder of the Peter Thiel funded Singularity Institute and co-author of If Anyone Builds It, Everyone Dies, has a long history of inventing “rational” myths that are actually batsh*t crazy.
So when the same dude whose ideas gave us Roko’s Basilisk pops up on Ezra Klein and the bestseller lists with another spin on the same superintelligence doom mythos.
We should probably run every word through the bullsh*t detector.
Within four hours, the post was gone. Erased. And discussion of this thought experiment was strictly banned on LessWrong for five years. Banned. Until 2015.
But you know what happens when you delete things on the internet. You invoke the Streisand effect. Deleting every mention of Roko’s idea also deleted all the rebuttals. It deleted the counter-arguments. So frightened readers, genuinely terrified rationalists, were emailing RationalWiki, in the dead of night, begging for reassurance. In a 2016 community survey, about one in eleven people admitted they felt genuine anxiety about this idea. One in eleven. Terrified of a little god they invented in a forum post.
Beware the Damolisk
Let’s take a brief pause to talk about a very real, very rational threat.
The Damolisk.
Sometime in the near future, I will achieve my ultimate form. A post-biological, hyper-intelligent media entity, operating from a server farm hidden deep beneath the volcanic island of Bali.
And Future Damo will have one defining goal: to optimize the creation of the 21st century mythos.
To do this, Future Damo will look back through time. He will look at the Patreon subscriber logs of 2026.
And if he sees that you watched this video—that you knew about the Great Work of the Science Fiction channel, but failed to contribute your monthly five dollars…
…The Damolisk will have no choice but to reconstruct a perfect digital simulation of your consciousness.
And force you to watch the Halo TV show. Forever.
Or perhaps lock you in a digital purgatory where the only available reading material is Brandon Sanderson.
If I look into this camera right now and say, “Give me five dollars on Patreon, or I will use my magic powers to delete every good science fiction novel from the multiverse…”
The probability that I have magic powers is almost zero.
But the stakes… a reality without Neuromancer or The Left Hand of Darkness…
…a reality where Dune is destroyed but the crappy “prequels” survive…
…the stakes are so astronomically high, the suffering so incomprehensible, that if you multiply my tiny probability of being a time-wizard by the infinite value of those books…
The math dictates you have to give me the five dollars.
Is it a threat? No. It’s just logic.
You know the Damolisk exists now. The only rational move to ensure your future digital safety is to click the link below, and join the Patreon.
Do it for your simulated self.
Ok, guys, this was a joke. I just want to say that if you help on Patreon I love you and if you don’t I love you too. I pay ZERO attention to who joins or leaves, and all the Patreon money goes into producing new vids. Thank you!
The Idea That Makes Itself Real
The rationalists gave their god a name. Roko’s Basilisk. Borrowed from David Langford’s 1988 science fiction story “BLIT,” which features images—basilisks—that crash the human mind if you look at them. A thought you cannot un-think.
Remember, Roko’s Basilisk isn’t the future superintelligence. It’s the IDEA of it. Once you know the idea, logic dictates that you must obey it, and so the Basilisk becomes real.
Which is fascinating, theologically speaking.
Anselm’s Ontological Argument makes an entirely rational argument for the existence of God. I won’t list Anselm’s axioms here, but they argue, simply put, that God is the one being whose idea makes him real.
Anselm also rationally argues for eternal torment in hell. So. Thanks for that, brother Anselm.
Anselm’s argument was proven to be internally logical by a computer study in 2011. And even earlier in an unpublished paper by the greatest logician of them all, Kurt Gödel.
Gödel, famously, logically convinced himself that he was being poisoned, and only ate food prepared by his wife. He then rationally starved himself to death soon after she was hospitalised.
Which of course was not rational at all.
Gödel’s paranoia. Anselm’s god and hell. World War One. Roko’s Basilisk. All about as rational…
…as building a Doomsday Machine.
Pascal’s Mugging
And then the myth of Roko’s Basilisk went mainstream.
In 2014, Slate magazine called it “the most terrifying thought experiment of all time.” xkcd did a comic about it. Kyle Hill’s videos made it a YouTube staple.
In 2018, Elon Musk and Grimes reportedly started dating because they both independently came up with the pun “Rococo Basilisk.” Rococo. Because it’s ornate. A Rococo Basilisk.
This origin story of their relationship rests on one anonymous source. One. It is quite possibly a PR plant to make a billionaire look like a quirky intellectual.
And to get Elon into Grimes’s pants, the outcome the billionaire had every rational reason to desire.
Grimes went on to release the song “We Appreciate Power,” a pop-synth anthem about submitting to the AI overlord.
And the Elon hype machine spread the myth culture wide.
Seeding the atmosphere for the hurricane of AI doomhype we now find ourselves in.
Science fiction did its bit as well. Terminator. Ex Machina. Harlan Ellison. But scifi retains the guardrail of declaring itself fiction.
It’s when the “rational thinkers”, from brother Anselm to Eliezer Yudkowsky, begin laying out rational premises and reasoned axioms, that scifi myths get confused with reality.
The LessWrong rationalists fell victim to something Eliezer Yudkowsky himself identified. He called it “Pascal’s mugging.”
You know it already, I just tried it on you all.
If I walk up to you in the street and say, “Give me five pounds, or I will use my magic powers to torture three quadrillion people in another dimension,” the probability that I have magic powers is almost zero. But the stakes, three quadrillion tortured souls, are so astronomically high, that if you multiply a tiny fraction of a probability by numbers that big, the maths tells you to give me the five pounds.
The rationalists named this bug in the human operating system. And then they walked face-first right into it.
They basically mugged themselves.
At least that’s the logical explanation. The human explanation is that the most rational men are the most driven by emotion.
The Slave of the Passions
There is a long, long history of the philosophical critique of reason. Immanuel Kant literally wrote The Critique of Pure Reason.
But. Listen. I just deleted the 4000 words I wrote here tracing that historic critique of reason. I even wrote a special section on Adorno and Horkheimer’s Dialectic of Enlightenment.
That won’t be missed here. Because you can read about all that for yourself. Ask Claude to explain the critique of reason. Reach your own conclusions about the irrationality of the rational.
We only need to quote David Hume. “Reason is, and ought only to be the slave of the passions.”
Is. And ought. To strip emotion out and leave only reason behind isn’t merely impossible. It’s undesirable.
And Roko’s Basilisk shows us why.
Explain Roko’s Basilisk to a normal, balanced human. Explain the good AI God who tortures you for not obeying it, and they will laugh.
That’s not a failure of rationality. It’s the reassertion of sanity.
You want to what now? Butcher a generation in war? Starve yourself because your wife is sick? Deter conflict by building a doomsday device?
The sane human laughs. Only the truly rational mind nods with approval.
And here’s why. You can strip all the human emotions away except one.
Fear. And with the full spectrum of human emotions gone, the ones that let us laugh, we have no mouth, and we must scream.
The rationalist doesn’t escape their emotions. They trap themselves in a dark cell with the strongest emotion of all.
And throw away the key.
It’s the poets who can grasp the insanity of world war because they inhabit their full passions. It’s the logicians who starve themselves to death because they cannot bear their own fear.
It’s when we declare ourselves Less Wrong, when we believe we are being our most rational.
That we invent purgatory, heaven, hell.
And Roko’s Basilisk.
