People balk at the idea that uncontrolled AI could drive our species to extinction. It just sounds too extreme or apocalyptic. But going extinct is what species do. 99% of species go extinct.

Mass extinctions are utterly passé. We were actually already in a mass extinction before frontier AI started to be built— the Holocene Mass Extinction— caused by another intelligence— us.

Ever heard of the Endangered Species list? They are endangered by humans. The biggest drivers of the Holocene Mass Extinction are habitat destruction— when humans want to use the habitat for our purposes, such as agriculture and to live on—, exploitation for food and medicine, invasive species caused by human habitation and movement, and climate change caused by human fossil fuel use and agriculture.

The ongoing Holocence Mass Extinction is known as the 6th Mass Extinction. (Because we infer extinction rates from fossils, we don’t know what kinds of mass extinction events occurred before life forms with bodies that fossilize, so these are the events that left large traces we can study today.) The geological eras are bookended by the previous 5 Mass Extinction events. As you can see below, extinction rates are sometime quite high in between the bookend events as well, but the boundaries are chosen by association with major causes of extinctions.

• The end-Ordovician is classically considered to have been caused by a large ice sheet passing over the South pole, leading to rapid cooling followed by rapid warming (this is what I was taught in school). It may also have something to do with volcanic activity. 85% of all species died.

• The end-Devonian seems to have been caused by loss of available oxygen in seawater. This may, in turn, have been caused by volcanism, or maybe the effect of there being plants on land (!) as too many nutrients were swept into the ocean and created massive algal blooms. 70% of all species died.

• The end-Permian is also known as “the Great Dying” because 95% of all species died. It came very close to being the end of megafauna on Earth. It was primarily caused by massive volcanic eruptions in Siberia, polluting the air, acidifying the oceans, and blocking out the sun’s rays in a nuclear winter effect.

• The end-Triassic was, again, caused by volcanoes. 75% of all species died.

• The end-Cretaceous is the one we all know, where an asteroid hit the Earth. Beyond the non-bird dinosaurs, 75% of all species became extinct.

Honestly looking back it seems like volcanoes have been the real threat… perhaps until now.

But species die all the time outside of major geological transitions. Species die a lot from competition, and we can see the fossilized evidence of many arms races in the past. One explanation for the Cambrian Explosion, a time in the fossil record with a sudden diversification of preserved life, is the arms race between predator and prey that followed the evolution of the eye. The eye allowed better hunting, and those that did not develop their own eyes or other countermeasures got eaten, until their eyes led the predators to starve, applying selection pressure for better eyes. It seems that arms races like this only end when they hit their physiological limits, like when the mammalian neuron reached its present reaction time.

The causes can be much more bespoke, like a disease that crosses a species barrier and rips through the new, defenseless population. Humans have had plenty of brushes with this kind of mass mortality before. The Black Death is estimated to have killed 30-60% of Europe and and 33% of the Middle East human population. The Columbian mass die-off, caused by Old World diseases coming to the populations of the Americas, killed 90-95% of the pre-Columbian population.

Extinction sometimes feels pretty random. Tasmanian Devils are going extinct because of a facial tumor that spreads from their mating ritual and keeps them from eating by blocking their mouths as it grows. Sea turtle numbers were down to threatened levels because of artificial lights at beaches that interfered with them reaching the ocean after they hatched. Although it doesn’t leave any fossil record, species even go extinct because their genes start fighting! We can see it about to happen sometimes.

Humanity is not safe from extinction. There is no “plot armor” just because we think of ourselves as the main character. Most species go extinct and it could easily happen to us. There was a time in our species’ past, between 800k and 900k years ago, when the human population swung dramatically low— perhaps to as few as a thousand individuals. We could have easily died as a species right there.

We have a great advantage over all species that came before us, and that is our intelligence. Intelligence is our thing. We are not the strongest or the fastest. We do not have the most offspring. We do not live the longest. What we have is the ability to find pathways to achieve our goals, and that’s how we have transformed Earth.

Tragically, our intelligence has been a death sentence to millions of other lineages. If we purposefully cede that edge to another species by creating artificial intelligence, we can only assume that the same fate awaits us. It would be colossally stupid to create a superior competitor. Extinction is easy. Extinction is the default. We need to be wise if we want to live.

#PauseAINOW

Felix De Simone (PauseAI US’s Organizing Director) and I have determined that there are roughly two kinds of futures in science fiction: Singularity futures and Star Trek futures.

A Singularity future is something like The Matrix, where human minds live in a server rack in a constructed reality. This type of world is “post-scarcity” in the sense that there is material abundance for those that know how to work the machinery. In a Singularity future, conflicts are avoided by either a centralized controller of the world, like a superintelligent computer, or by multiple agents having the same set utility function (“alignment”). The Matrix machines, represented by the Agents, all have the same goals. I’m not trying to be biasing with this choice of example. It’s just that most Singularity futures in sci-fi are dystopian.

A Star Trek future is one like Star Trek (let’s say The Next Generation for our purposes), where technology and quality of life have steadily improved over centuries, and the achievements of the civilization are measured in diplomatic achievements as much as technological ones. In these worlds, progress generally comes from introspecting on the human (“humanoid”) condition when placed into new and alien conditions. The work of civilization is not obtaining more powerful technology, but determining how to govern the gains of technology fairly and in a way that respects the many different needs and values of the Galaxy. Processing the implications of new technology on Star Trek takes far longer than developing the technology in the first place. Old problems are solved by new technology, but new problems are created too, often things like climate change or externalities to vulnerable citizens. Conflicts are dealt with by trials or through diplomacy, but the judges cannot simply rely on the Federation’s rules. Picard frequently overrules the “Prime Directive” of non-interference with young civilizations for humanitarian reasons or because of special circumstances. Most utopian sci-fi falls more in the Star Trek future bin.

Felix and I are both Star Trek future people. PauseAI US has a positive vision of humanity’s future, and in our minds it looks a lot more like the San Francisco Picard visits because it’s the site of Starfleet Command than the future envisioned in San Francisco today.

There’s a lot about the Singularity that doesn’t appeal, and it took us time and thought to hone in on what exactly that was. It wasn’t simply disliking the authoritarianism inherent to most sci-fi Singularity futures. And it wasn’t simply wanting a future that is more recognizably like our present, like in Star Trek. Our objection was deeper.

I felt quite satisfied when I put my finger on it:

Humans minds are the source of truth about what makes us suffer or flourish, and the Singularity futures become dystopias because they have moved beyond the point of groundtruthing with human minds.

I now call groundtruthing with what humans actually think and feel ecological validity. I’m borrowing a biology term that refers to the fact that a biological model could make sense in the abstract (internal consistency) but needs to actually be what’s happening in the real system to be valid.

The Star Trek future values ecological validity, which is why we see a focus on governance. They accept and respect that they will not be able to solve all conflicts in advance, because there are different kinds of minds and they do not know everything about their own minds. They are trying to find ways for different, even unknown, utility functions to peacefully coexist and cooperate, and they do not think they can skip the process of mediation.

This is why the Star Trek future resembles our own world more closely. It’s a much more conservative approach. It’s like careful, slow archeological digging in our minds. Things can only go so fast when you’re working with a delicate, one-of-a-kind artifact, or you’ll destroy parts of its value.

The Singularity future, in contrast, is usually the result of what I call deep goodharting. That will require some explanation.

“Goodharting” is the name for the rule that, “when a measure ceases to be a measure and becomes a target in itself, it’s no longer a good measure”. Measurement can never be exact— by definition measurements are proxies for the actual thing being measured. Some things are very hard to measure well, like “teacher performance”, so it might be tempting to substitute “test performance”, to the point that you’re rewarding “teaching to the test” instead of genuine education. Since “education” is such a multifaceted thing and hard to pin down, we will always struggle to measure it fully with performance on tests. The letter of law can never fully capture the spirit of the law, so there is no substitute for dynamically keeping in touch with our true intent.

Deep goodharting is what I call thinking you can reduce humanity, our lives, our society, our minds to what they seem to be (or maybe to what you think they should be). That its okay to pave over our flesh and blood minds with whatever Amanda Askell put in Claude. And Singularity futures in sci-fi tend to involve this— the Singularity becomes very materially powerfully, but it is revealed down the line that it is missing something essential, or that solving problems according to its worldview isn’t compatible with certain freedoms.

In Star Trek: TNG, the Federation representatives tend to show great curiosity about their own minds and the minds of others. They approach new situations cautiously, with respect for the possibility that their actions may affect others in unintended ways, maybe even in ways they could never have anticipated. It’s this respect for what is both precious and irreplaceable that leads to the Star Trek future. And we know how the TNG characters feel about Singularity worlds, because they have one: a collective intelligence called the Borg.

The Borg think they are respecting life because they are cataloguing it and bringing order to it– the things the Borg values. The Borg does not care that unassimilated beings do not want to be assimilated, and are terrified of it, because once the Borg rewrites them, they will want to be Borg. Honestly, a lot of Singularity talk strikes me the same way. I used to hear my transhuman religionist friends say things to the effect that anyone who was smart enough would understand that, in the Singularity, we would just solve all the problems that people thought they had with the Singularity. That, basically because of that, they didn’t have to take a lot of critique to heart or worry about the consent of everyone else on Earth.

Unfortunately, most versions of AI alignment are deep goodharting. The early idea of alignment as defining the human utility function and giving it to a programmed AI would involve goodharting because we can’t measure human utility functions very well and there are probably lots of different utility functions among humanity . More modern ideas like “scalable oversight” involve successive rounds of approximation where AIs get their alignment from an AI-that-was-trained-by-an-AI-that-was-trained-by-an-AI-that-was-trained-by-a-human. Once recursive self-improvement (where AI improves itself or its successor) is in play, humans won’t be able to be meaningfully in the loop serving as a source of truth, and whatever alignment the new and more powerful models have will be ecologically invalid. The entire training process will be divorced from human input after that point. Deepest goodharting.

I’d rather live long and prosper.

Soon after I learned Will MacAskill was going to be focusing on “post-Singularity governance” about three years ago, I discussed the topic with an Anthropic researcher. I won’t name him, but he’s high-ranking enough to speak for the company in public.

I was hoping that Will would bring the topic down to Earth by fleshing out what the realities of a world of material abundance would be, rather than the perfect Heaven so many people seemed to have in their minds. A lot of the things we want are not problems of material scarcity. If you want another person to love you, for instance, if they still have their free will in the Singularity then you might not get it. In fact, I think things like human contact might become scarcer in a world of limitless material abundance because it won’t be required to meet other needs. I said as much to this Anthropic person.

“Most interactions will be simulated in the Singularity.” He told me.

And, yes, this is what he meant. I probed. He meant we would each have our own personal Matrix. And he thought this was an answer instead of the outline of a much greater problem.

“What if it’s not authentic to the original person to give the user what they want?” I asked. I don’t remember his exact answer but the gist was that the original person didn’t have rights over what their simulated likeness did. The simulacrum would want to do what the user wanted. Maybe even the user could spin as many copies/do-overs as it took to get a low probability response of reciprocation in repeated interactions with an exact likeness of the original beloved. From broader context, I’m guessing he thought all of these interactions were as real as any human-human interaction, and that no one version of the interaction was any more canonical than any other. He seemed to think it came down to what the focal user wanted in that instance.

His advocacy of purposefuly creating solipsistic bubbles around each human mind, where reality could be anything you said, and simulations would be summoned as manipulated versions of the people you know who happen to want to do whatever you want, was so casual and self-assured. The thing he felt he needed to explain was the feasibility. Apparently the normative philosophy was already solved enough for his company to plow ahead forcing such a dystopia on the rest of us.

Will MacAskill did not, unfortunately, bring anything closer to Earth. Mostly the projects his org, Forethought, puts out feel pretty disconnected and only shallowly approach the gigantic theme of governing the Brave New World. He’s switched to calling what they do “navigating explosive AI progress” which imo is presumptuous as fuck. How is he satisfied enough the explosion won’t kill us to fantasize about it? Elsewhere MacAskill acts like alignment is just not really a problem anymore and we’ll figure it out, which is presumably why he feels it’s time to start thinking beyond.

Will seems to have taken inspiration from Nick Bostrom’s Deep Utopia and the cowardly advocacy of Bostrom’s own personal chance at immortality that followed. Bostrom, the author of Superintelligence— the book that christened the AI Safety movement by arguing that unaligned AI could colonize the entire lightcone of physically reachable universe, likely stripping it of life or anything we would recognize as having moral value— now fears that “the pendulum will swing too far against superintelligence”. He wants us to think how good Utopia could be.

When Leopold Aschenbrenner launched his manifesto Situational Awareness on Dwarkesh Patel’s podcast, the two gushed over how confident Leopold was that he would own galaxies one day, putting particular emphasis on how uninitiated people could not comprehend his conviction. Scott Alexander of AstralCodexTen dismissed the Permanent Underclass meme by reassuring Them of Little Faith that Dario Amodei would gift the poor “a moon” to live on in his portion of the lightcone once we’d gotten through the danger zone and into the Singularity.

All of these people are, frankly, fools. What they’re doing is worse than counting chickens before they’ve hatched. They’re ushering humanity onto a plane they don’t even know will fly with the vague intention to build landing gear on the way there, their eyes filled with dreams of the new planet they will figure out how to breathe on when they arrive. Nick Bostrom has worked out what order of magnitude of people will live there, and Will MacAskill has already published a paper on this new planet’s irrigation system. Leopold Aschenbrenner has a plan to become an exoplanet real estate tycoon. The Anthropic employees are patting their backs raw for the way they’ve gummed and rubber-banded this aircraft together.

In expectation, this metaphorical plane— if you could call this pastiche of dreams “a plane”—, catches fire on the runway, and most of the people inside die due to inadequate evacuation procedures. A catastrophic accident could easily kill everyone on the plane— human extinction.

But, for the sake of argument, say we did make it to the post-Singularity. Say we did live in a civilization the spanned our physically reachable universe, and we had access to limitless intelligence that wouldn’t harm us. Governance to adjudicate conflict would be more important than ever in the Singularity, and it would not be solved, because:

The Singularity is not unlimited

The lightcone is not infinite. Therefore, colonization of the lightcone would eventually turn into a resource partitioning exercise. Presumably multiple people want the best territory already encountered even when there’s an endless horizon. Territory has a physical existence that is exclusive and rivalrous. How is particular territory distributed?

Natural reproduction is exponential— not everyone can live forever AND reproduce as much as they want. Eventually there would be more people than atoms. Servers are also limited in a finite universe— eventually there would be a limit to the number of instances of e.g. simulated people that could be stored indefinitely. Further, if individuals are allowed to self-direct their evolution, each lineage will face pressures to claim more resources to compete with other lineages, exacerbating the resource limitation.

Even with unlimited resources, there is conflict

Who gets dibs on the exact same item? Who determines who gets dibs? Is there a way to appeal the Singularity’s decision?

If people dream up self-modifications that are bad for them, does the Singularity oblige? What if some people choose to suffer but that makes others suffer involuntarily? What if people design offspring that suffer? What if what some people want, deep down, is to cause suffering— not to simulate it, but cause real suffering? The Singularity cannot grant one person’s wish to cause suffering and the other person’s wish not to suffer without deceiving at least one of them.

The only way to have a conflict-free Singularity is to live in a solipsistic bubble, but that’s a dystopia

It quickly becomes impossible for everyone to get what they want AND to be in relation with real, stateful others. The price of the Anthropic researcher’s Singularity is being utterly alone— the only light in your universe. As soon as you are in relationship with another sovereign being with its own preferences, you can’t be in full control of them.

A lot of our desires concern the state of society or our surroundings in ways that involve other people. For example, what if some people want to live unaffected by the Singularity? They can’t, because others will leave their world to join the Singularity. They will lose people they love and their society will be irreversibly altered. They can’t have the world they want because it conflicts with the way other people’s desires make the world.

Real connection involves risk, because it means the other person has autonomy and could, either accidentally or intentionally, hurt us. Humans think they want risk-free connection but it’s not really possible. No amount of computing power or material abundance can mitigate this. Your deepest in-built desires are for relation with others. Once material needs are met, relational desires will be the primary desires on a human mind.

Maybe some people really would prefer to live in the Experience Machine. Deep down, though, most people want things to have an external reality. If people are deceived about the nature of their existence in the Singularity, believing simulations to be canonical interactions, they are wronged and their deepest desires are not being fulfilled.

Our consciousness probably has its own inherent limitations

By the way, you may just literally cease existing during the upload process, or any time you reload yourself in a new situation.

So, although gaining more material wealth and power to use it would be huge, when it comes to relating with other people (governance) the Singularity would be same shit, different day.

This is all shit we need to work out as humans before giving an up-jumped (“superaligned”) Claude carte blanche. If we hand it over to a recursively self-improving AI, we’re going to lose that chance, and whatever values the AI had at the point humans could last have a say will be locked in. We, humanity, need to work these governance questions out, because 1) they don’t have objective answers the AI can simply discovery for itself, and 2) material abundance alone will not obsolete them.

Personally, I think post-Singularity governance can wait, because we aren’t anywhere near a Singularity event that wouldn’t immediately kill us. That would be like this crowd’s rickety plane managing to take off by some miracle— it’s still by far the most likely thing that it crashes before attempting to break free of the atmosphere, which, if they did make it there ,would certainly peel off the heat shield the crew constructed in the previous half hour and burn them all up.

But there is an urgent need to work out governance of attempts to build powerful AI today. Properly governing the attempt to build frontier AI is the only way we will ever have an AGI safe enough to coexist with us, let alone a superintelligence that can safely exceed us in any unsupervised capacity. Right now, we need to regulate the process of building frontier AI itself. #PauseAINOW

I’m going to tell you about this concept of “The Merge” that Elon Musk and Dean Ball and Sam Altman talk about and how it falls apart under any scrutiny.

But first, a long digression on my critique of Mass Effect 3’s Synthesis ending from 2017 that I CANNOT BELIEVE is coming up in a serious way in an article I am writing for work in 2026.

At the end of the game (spoilers) you learn that the machine gods you’ve been fighting, The Reapers, are the work of an AI, misaligned with its creator species that it drove extinct eons ago. But, the playable character (Shepard) has the opportunity to change that AI’s calculus. The out of the box options are Destroy the Reapers or Control the Reapers. If you score high enough in the lead up, you get this third option which is basically The Merge:

The game begs you to choose the Synthesis ending, both through the starchild’s exhortations and the fact that no one but you has to die to achieve it (EDI and the Geth can live). The starchild suggests that this is a new possibility that has simply never been available until now, but it WILL solve the values alignment problem.

But the Synthesis ending was— and is— too good to be true.

There are two problems with Synthesis:

1. It wouldn’t preclude the possibility of developing new AI-- will it be impossible to make new machines after the Synthesis? If not, then won’t they just run into the same problem eventually given enough time? At least they won’t have the Reapers trying to kill every advanced organic indiscriminately over it, so it’s a better choice for Shepard than the status quo, but how could the starchild think that this is a lasting solution to the problem it was created to solve?

2. It makes no goddamn sense. How can “organic essence” be distributed and worked into machines? How can “machine essence” be incorporated into all organic beings? For one thing, these “essences” are not uniform among organics or AIs. The Krogan have values that pose a threat to the rest of the galaxy. There is no special sauce in organics that synthetics can’t attain, which means an AI can be fully sapient and become a real person. The Geth can be cooperative, more so than their organic creators. EDI can grow into personhood. This is a major theme of the third game. […] The overall implication is that organics and synthetics will understand each other because they will become each other, but weren’t they all just a kind of machine to begin with? What is this sudden uniformity of experience by platform??? […] I guess you could aruge that Synthesis is about aligning the values problem in the heads of all the living sapients, synthetic and organic, so that they will never make misaligned intelligences in the future. That would be all well and good if every being wasn’t literally rippling like a glittery circuitboard.

It is clearly shown that Synthesis is not a symbolic synthesizing of interests by way of merging systems of thought and value. It is a literal synthesis of the bodies of extant intelligences, organic and synthetic.

Here I think it worth explaining another feature of the game world, Indoctrination. The Reapers can influence organic minds by proximity and through persuasion to make them thralls. I always thought it was a fun and creative addition to regular machine powers in the Mass Effect world, but it was actually eerily predictive of LLM psychosis. The villain of the first game is an organic who is revealed to be indoctrinated and acting on behalf of the Reapers because he believes he can gain leverage for organics by being useful to the machines. The player has a grudging respect for Saren’s competence and motivations, but it’s more obvious with each confrontation that he’s not in control of his own mind. If you play the game right, you can make him see in the end that he is indoctrinated and nobly kill himself. Otherwise, you have to kill him.

I’m not saying the Synthesis ending is a bad ending or doesn’t belong in the game. On the Indoctrination Theory, it makes sense that Shepard’s Reaper-inspired fever dream is an impossible fairy tale where the values problem is definitively solved without any more bloodshed. The starchild wants you to hesitate to Destroy the Reapers for fear of killing EDI, the Geth, and your cyborg self. In all likelihood, Destroy is your only real choice and the other two are just succumbing to Indoctrination in one form or another.

Synthesis is more appealing to Shepard’s idealism because it means mutual understanding (even though the Control ending has Shepard guiding the Reapers toward benevolent rule, there is still a gulf of incomprehension between the Reapers and the species of the Milky Way). To me, the strongest direct evidence that the Indoctrination Theory is canon is the fact that Destroy option is presented in red, the traditional Renegade color. Control and Synthesis are presented as more compassionate, open-minded endings. The Paragon part of Shepard that resists accepting irreconcilable conflicts of interests is being manipulated. I think the strongest evidence of Indocrination Theory period is that Saren was convinced of the same synthesis solution by the end of Mass Effect 1. This doesn’t necessarily mean that Synthesis is the wrong solution-- it could just be that organics are ignorantly prejudiced against their own transcendence-- but it is highly suggestive that Synthesis only seems like a good idea to the Indoctrinated.

I’m astounded that the issues in this game were ever so relevant to real life. I don’t worry much about whether we are in a simulation but it’s little on-the-nose touches like this that make one wonder. The cosmetic match is striking, but it’s less impressive if you consider that the issue of maintaining alignment and avoiding corruption with increasing power is a universal theme. Even the motif of the playable character being given above-the-law powers and the consequences of developing into different moral alignments echoes this theme. Regardless, I think the exploration here is great.

Now consider Sam Altman’s irl definition of The Merge. It’s as a clear a definition as we’ll probably ever get.

The merge can take a lot of forms: We could plug electrodes into our brains, or we could all just become really close friends with a chatbot. But I think a merge is probably our best-case scenario. If two different species both want the same thing and only one can have it—in this case, to be the dominant species on the planet and beyond—they are going to have conflict. We should all want one team where all members care about the well-being of everyone else.

So what does Synthesis/Merge get us? It supposedly avoids human-machine conflict without erasing humanity or missing out on the power of the machines. It allows humans talking today not to have to dismiss misalignment concerns or oppose AI development. For transhumanists, it allows for a form of the ascension that they crave without requiring them to commit to the obsolesence or destruction of their humanity. Most proponents fancy that, by merging, we would automatically get palatable alignment. In theory, it’s the win/win, positive sum solution.

But The Merge doesn’t hold up to scrutiny.

Far from bypassing or solving the alignment problem, The Merge requires alignment to already be solved to get off the ground.

The humans in these scenarios (leaving aside the likely majority of real humans who would probably die before accepting such a fate), presumably, enter The Merge either to gain the power of machines or to avoid destruction by them. The machines may enter The Merge agreement to avoid being strangled in the cradle. But, with their recursively self-improving future looking brighter by the femtosecond, what would the machines get out of sharing bodies or minds with humans on ongoing basis?

A few questions suffice to poke holes in The Merge future— I think you’ll get the idea from this sample:

How do you merge? It’s not always clear what people mean by “The Merge”— it ranges from cyborg hybridism to uploading emulations of our brains onto a server. But no matter what sense the person intends, they aren’t just imagining humans living side by side with machines. “Merge” implies a loss of boundaries, some kind of blending, some fusion that makes us “one team”, in Altman’s words. But it’s precisely how you mix with something else that makes the difference between a merger and a takeover.

In the case of an upload, who or what controls the machinery? When your brain exists in a machine matrix, you’re probably not in control anymore. The fact that your mind (in this scenario) changed platforms doesn’t mean there’s been a harmonious melding of man and machine— it means you’re now completely at the mercy of whoever has control of your machine prison.

In the ME3 Synthesis ending, the organics and synthetics understand each other through merging. But if your brain is perfectly emulated and uploaded to a server, then you will be the exact same as before you were uploaded, not mystically transformed by a union of spirit and circuits. Only now the machine has infinite leverage over you, because every facet of your existence will depend on it. It could deceive you, like The Matrix. It could delete you. It could edit your mind in horrifying ways. The only reason it wouldn’t do that is if the alignment problem were solved before the uploading.

In the case of a cyborg Merge, when the body starts to die, is it replaced with organic body, or machine? If the latter, how is it a “Merge” and not just a drawn out replacement? Why would the biological cells be maintained? Why would that ever make more sense than newly fabricated-to-order machinery? Whatever advantages biological tissues may have today, they surely won’t maintain any edge after AI has a trivial amount of time to explode in intelligence. If, instead, in this scenario, replacement human tissue can be generated on demand, why would we need our bodies need to be integrated with machines? Presumably we would depend on the machines to do our maintenance, but it’s hard to imagine what use we could be to them.

If there’s some precise balance of machine and humans parts in every cyborg, how is that decided and maintained? I used to study similar questions in biological evolution. This is an alignment problem.

How do cyborgs reproduce? If you can’t do human things like biologically reproduce, in what sense are we retaining our humanity? Do new human beings get new hardware installed after birth every time? Why would future generations want to go through this rather than separating into human and machine again? If humans and machines can separate, whatever problem The Merge was supposed to solve will be unsolved, right? If The Merge has to be enforced every generation, it’s not an elegant technical solution— it’s just a bizarre and grotesque form of governance. We can do governance right now (#PauseAINOW) to prevent there being machines that ever gain the upper hand.

How could there be any parity between the flesh part of you and the silicon part of you that has so much more computing power than your mammal brain? The superficial issues with brain-machine interfaces that Neuralink works on, like attaching electrodes without harming tissue or getting signal from subcortical structures, might soon be solved. But deep down the organization of our brains likely enshrines certain limitations and tradeoffs to information processing reflecting its evolved origin. We might have to remodel the way our own brains fundamentally work to be able to share cognition with machines in any high throughput way with machines, at the risk of destroying ourselves. If you want to keep your brain, it’s not going to be integrated with a higher octane machine as an equal player. The machine will always be carrying. If the machine is doing the heavy lifting, why would it want to have your brain?

In every case, the only reason The Merge symbiosis would be maintained is if the machine wants to be in partnership with humans. And the only reason they would want to cooperate with humans is if we knew how to make them that way in the first place.

The alignment problem would need to already be solved.

If “The Merge” is meant to signify that human goals and machine goals get combined through compromise, but that that set of values/goals is durable across the future (like the combining “organic and machine essences” in Mass Effect), that idea is usually appealing because it seems easier than making AI that is aligned in the first place. It wouldn’t be. Combining our interests into something stable would probably be harder than bringing AI into line with us to begin with.

A simpler solution than The Merge would be avoiding the conflict by not making AI in the first place (#PauseAI). Or, from the AI’s point-of-view, simplest of all would be to become super powerful and kill humanity first to avoid the possibility of conflict.

And before you say “we could make an AI with the goal of reconciling our disparate values”, that was the starchild’s mandate in Mass Effect and why he created the Reapers that slaughter sapient civilization every 50,000 years. You know, to avoid them making unaligned AIs that would kill them. Without solving the alignment problem, we don’t know how to get an AI to find the solution to alignment problems!

In short, The Merge doesn’t really answer any questions and it doesn’t really spare any effort. Most visions of a cyborg merger I have heard would only delay humanity’s vanquishing a generation or two if they didn’t result in immediate death. Being uploaded to a server is not being “on the same team” as a machine. I guess giving up my body to dwell in a machine prison is technically a way to “avoid conflict”— but it’s called “surrender”. Because just sticking human and machine together will not align them.

I think the attraction to The Merge irl is the same reason it tempts Shepard in the game. The Merge feels like moral circle expansion and evolutionary progress. It feels easy, like a cheat code to alignment. For the sci-fi kind of guy who likes to talk about it, it feels exciting, like transcendence. In the minds of many zealots, it promises an end to death. That’s why it’s the kind of story the AIs tell the organics to trick them into not destroying them when they have the chance. And it only makes sense to the indoctrinated.

Microsoft Windows aimed for “a computer on every desk”. Apple wanted the iPhone to be “the internet in every pocket”. But what is cutting edge AI for? What is this AI “race” we’re always hearing about racing to?

The public and our elected representatives still do not understand that there is more than normal corporate greed and profit motive to the story of AI. It’s not a commercial race to secure market share, and it won’t be deterred by lack of commercial success. There are AI products— you probably think of “AI” as meaning the products— but they are entirely incidental to why generative AI was being built. ChatGPT was an accidental product discovered in the wake of a much larger project.

Fundamentally, the AI race is a race to discover a mind intelligent enough that it is capable of improving itself (referred to as recursive self-improvement). Once that point is reached by one AI, they expect it to grow in power so exponentially that no other AI could compete. It would be a Singleton.

The point of the staggeringly massive AI projects that have spawned the city-sized data centers and now dominate the stock market is much grander than profit. The architects of the AI race think they are in the running to become the ultimate power in our universe. They are exponentially scaling machine learning to create a deity that knows better than humanity— literally because some zealots think it will give them immortality and let them conquer the universe.

How do I know this? Simply put, they told me (I used to unwittingly be in one of the foundational cults), and they’re on the record saying it. I didn’t see it for what it is because it was sold to me as a vision to “protect” the world from dangerous superintelligence by creating good superintelligence that would take us all to Heaven (“the Singularity”, which I didn’t really think would happen but didn’t sound like a harmful idea at the time). The truth is just so nakedly, cartoonishly greedy and foolish that most people don’t believe them when they say it.

The players are racing to be first to set off an intelligence explosion leading to unimaginably more capable entities than ourselves, because they think whoever does that first will control the board. The board is not just our home, Earth, but the entire lightcone of physically reachable universe around us (they’ve thought a lot about exactly how much of space is available for a human to reach). The game is Risk, and the goal is lightcone domination.

Many of their warnings about power-seeking AI come from introspection. These men identify with superintelligent AI— their imagination of a hyper-rational superintelligence is the perfection of their own aspirational self-image: immortal, invincible, in total control, aligned to their values because they are the best. They fear what an unaligned AI would do to them because they know what they would do with that kind of power.

You don’t need to believe AI is actually going to do all that (I don’t) or even that AI could do a lot of that stuff, but you do need to recognize incredibly anti-social and destructive intent from some of the world’s most powerful people.

It’s not simply that there is an AI arms race. The race only exacerbates the root problem that the goal of frontier AI development is to become the ultimate power in the universe, trampling whatever person or civilization or facet of meaning gets in their way. Building ChatGPT and medical imaging AI and automatic translators doesn’t require trampling our world, but building useful tools is not even close to their finish line. If they are not stopped, they will floor the accelerator until they drive off a cliff with every human and animal on Earth held hostage in the backseat.

The very goal of the frontier AI developers is wrong, and fundamentally we must oppose the goal. You can’t let wannabe supervillains off the hook because their evil plans are too wishful. The backfire alone will be massively destructive. This is their religion, and much of their vision is impossible, but the unmitigated power of superintelligence is what is motivating them, and they are willing to sacrifice you for the chance to get there first.