AI Alignment, Immortal Deity-Kings, And You
(Reviewer’s Note: There are spoilers in this review, as you might expect when reading a review of a 40-year-old novel. It’s also the fourth book in a series; I’ve done my best to make it readable to anyone, but I’m not going to pretend a baseline familiarity with the series’ setting wouldn’t help in reading it.
With that said, the book itself has very little in the way of plot; it’s a snapshot of a moment in time, a description of the mindset of an unusual ruler, and a consideration of the existential dangers that surround humanity in this fictional future and perhaps also in ours. I promise I won’t leave you entirely hanging; the important bits necessary to understand the conclusion are sprinkled throughout the review.)
The memory of sand’s gold sheen
The worm, the man, the Arakeen
The beast, the wise undying king
His long and gentle wrath
His voice trapped under golden swells
Like screams wrung from uncounted bells
divided god within a hell
His pain a golden path
- From The Collected Songs of The Scattering, author unknown.
As God Emperor of Dune begins, our attention is immediately drawn to people. Here, 3500 years after the chronological setting of the first novel, is immediate proof that humanity has survived in the form of a small group of people fleeing through a forest, wolves nipping at their heels.
The wolves belong to Leto Atreides II, the grandson of Duke Leto Atreides and son of Paul Muad’ib Atreides, the Kwisatz Haderach and protagonist of Dune I: The One You’ve Probably Read. At the end of the third book, Leto fused his body with Arakeen sandtrout, the larval form of the Sandworms on which the plot of the series mostly hangs. This symbiosis gave Leto super-human physical powers to match the clairvoyance already enjoyed by his family and allowed him to seize control of the galactic empire.
Centuries of time have seen him evolve into a hybrid of a human man and a full-grown sandworm, and the resultant power and pseudo-immortality have allowed him to extend his father’s dominance of the known universe from a period of decades to an era spanning the better part of four millennia.
The wolves are his not only by right of ownership but also apparently by right of design and creation; near-immortality leaves one with much time to tinker, and he has developed the wolves to a level of sophistication sufficient that they understand the boundaries of their hunting grounds to stop at the Idaho river. It is towards this river and the safety attained through its crossing that the group is fleeing.
The members of the retreating troupe also belong to Leto, although they may not acknowledge or even suspect it. The nature of his ownership of them is in many ways similar to his sovereignty over the wolves that guard the forbidden fortress around his citadel; they too are to some extent beings of his creation and design.
If this description has made you expect an action-filled novel, dispel those hopes. This is a book about the nature of leadership as practiced over periods of time that defy normal concepts of mortality. As a story, it doesn’t amount to much. As a picture of a fully-imagined empire at a particular point in time and a description of an unusual method of rulership, it’s something more. This is the least readable of the Dune books, but also the most meaningful.
It’s a book that meditates on the loneliness of true superiority, of the ennui of true invulnerability. It is a book that ponders humanity as brought to heel at the side of a benevolent and less-than-human mind with greater-than-human capabilities.
As was the case in the nearly two-Christianities-ago day of Leto’s father, control of the spice resources of the planet Arakis amounts to de facto control of the universe; it’s the unobtainium does-everything-you-need resource the dune universe runs off. The book likens this to Hydraulic Despotism, where control of an all-important resource (generally water) allows for control of everything downstream of and reliant on that resource.
Leto’s grasp of the resource that grants this near-universal control is unshakably strong compared to his father’s. He has driven the sandworm to extinction; he is all that remains of the species. What spice remains in the universe exists only in stockpiles he created over the course of his rule.
From those stockpiles he ekes out poisoned riches; a dose of spice essence to create a Bene Gesserit reverend mother here and a supply just great enough to enable the spacing guild to allow such travel as Leto approves there only serve to reinforce the universe’s reliance on Leto’s largess.
Leto’s prescience, his nigh invulnerability, and his unrivaled levels of experience both personal and inherited1 allow him to use this control to its utmost while never being much in danger of losing it; any assassination attempt would have to be of a magnitude sufficient to threaten Shai-Hulud, and Leto (for the most part) stays a safe distance from large bodies of water.
The book is thus a 400-page contemplation on the nature of power as exerted by a mind created by humans which has nonetheless surpassed them. It’s the story of an entity that seizes control and pursues paths and goals that seem right to it, but that are not necessarily in concert with the will of humanity as a whole.
Though likely unintended by Herbert — whose fear of mechanical thought took on a different flavor — it is impossible to fully ignore the echoes of AI risk throughout the novel. In reviewing the book for a 2020’s audience it’s necessary to return to that well again and again, to share dreams of humanity in chains, destroyed or kept in comfort by an entity with powers that exceed our own.
Leto II functions as both a person and a sort of societal force in the book, and this description only concerns his person. While connected, his function as a mind that emerged from humanity and controls it is distinct from Herbert’s personification of him as a man.
In the sense that he can be understood as human, Leto is a man on the brink of a mental collapse that never quite comes; the isolation brought on by being both in a position of untouchable power and a not-quite-human shell wear away at even the strongest of wills, it seems. Heavy is the head that wears the crown, and Leto has worn it for 3500 years.
This loneliness synergizes with Leto’s immense boredom, an ennui enhanced by experience and near-omniscience. Very little is new to him and virtually no one in the universe can serve as a companion to him. This makes him vulnerable not only to the psychological instability his loneliness encourages but also to anything that could possibly satisfy that lack; he would (and does) tolerate large amounts of risk to try and fill these gaps.
Perhaps the most dangerous aspect of Leto is his firm, unshakeable belief that he is doing the right thing. In Herbert’s universe, the rightness of his actions is not just opinion but instead verifiable fact, at least in the sense that Leto himself can verify them to his satisfaction.
This means that there is nothing Leto will not do if he perceives it as necessary; there is no cruelty he would not enact to push forward his golden path. But he is a mind that emerged from humanity and sees things humanity cannot itself see. His good deeds are not understood, and he works quasi-endlessly for what he considers to be the ultimate good while bearing the burden of the knowledge that he is viewed as nothing more or less than a dictator, a personification of the restriction under which humanity itself chafes.
In the second and third books of the series, Duncan Idaho returned to life (and death) in the service of the Atreides. In the Duniverse, both your mind and your cells have memory, each Duncan clone carries the memories of the original. Through the same mechanism of ancestral/cellular memory, Leto has seen Duncan’s sacrifices through two generations of eyes in addition to his own.
Faced with this loyalty, a sense of personal debt and the ability to order unlimited clones, Leto shrugged his shoulders and decided to always have a Duncan around to shoot the shit with.
From what we gather, the Duncans themselves always have very mixed feelings about this and tend to go slightly crazy with doubts related to how each of their predecessors met their end.
Duncan himself (or, at least, this iteration of himself) has misgivings about the totality of Leto’s control that clash with his absolute loyalty to the surname he serves:
”You’ve taken something away from us,” he said. “I can feel it. Those women… Moneo…”
Us against you, Leto thought. The Duncans always choose the human side.
Duncan is also faced with the reality any legendary swordsman faces 3500 years into a focused human-improvement breeding program faces: he’s slightly obsolete. People (read: women) generally like him and a giant worm-man loves him, but he seems to feel he lacks a clear purpose. He is a man who has always been useful wallowing in sudden, shocking impotence.
This combined with other frustrations lends his character a tone of someone casting about for something significant to do; he is almost from the first waiting on a match that can light his fuse and force him into meaningful action of any kind.
Remember those people running from the wolves at the onset of the novel? They were led by Siona in a successful effort to steal some intel from Leto’s own backyard. Siona is a rebel; she detests Leto and Leto’s rule. She will do anything to make even small progress towards hurting him, including sacrificing the lives of her friends, which was the eventual outcome of everyone who followed along on her lupine 5k theft-run.
She is unpleasant to most people, most of the time. Here is the nicest thing she says in the book, spoken directly after she kind-of-sort-of kidnaps Duncan Idaho to try to hurt his feelings:
“It would not offend me to be your friend,” she said.
It might sound as if she’s a bad character, but like most of the other major players she is more representative of a facet of humanity than she is a fully fleshed human being. She is humanity, developed, advanced and unconquered. She is as refined as her father, with none of his domestication.
You might occasionally hear someone talk about Ernest Hemingway not being able to write women - that they exist purely for the male characters in his book. They have not met Hwi Noree, a character who is literally designed to like a particular housebound rationalist utilitarian and who has no other characteristics whatsoever.
I’d like to write a whole bunch of insightful things about the character, but there isn’t anything to delve into. She’s from IX, a planet that builds technology that can mostly block Leto’s psychic powers, and was designed in a joint effort involving both the techy Ixians and the genetic meddling of the Tleilaxu.
And designed she was, for one purpose and one purpose only: Being attractive to Leto. The book describes her as “the epitome of good”, but “good” in this sense seems to be being from the start completely and unjustifiably committed to Leto; from her first day on the job as the Ixian ambassador to the god-emperor, she ruthlessly sells out her bosses and tells Leto everything he wants to know in just the subservient way he likes to hear it.
Lest you think I’m exaggerating, consider Herbert’s description of Hwi after Leto contemplates the difficulties of worm/human romantic relationships, becomes frustrated, and sends her away:
She left quietly, but Leto could see that Hwi, too, was tortured. There was no mistaking the deep sadness in her for the humanity that Leto had sacrificed. She knew they would have been friends, lovers, companions in an ultimate sharing between the sexes.
At this point, she has known Leto for approximately five minutes.
Besides being very nice to people who haven’t earned it from her (this is almost entirely aimed at Leto, but comes up in one other way later) she has essentially no personality traits. As a plot device, she exists only to be tempting to Leto, to be something he wants enough to compromise his primary objectives.
Herbert dug deep and tried to figure out what the most appealing woman possible would be, and came up with a personification of subservience and unearned affection so blatant it makes love interests from harem anime look like Susan B. Anthony in comparison. Lacking any kind of a personality or desires of her own, Hwi Noree exists almost entirely as a desirable object - a particularly shiny apple would have done just as well in her place.
If Duncan Idaho represents the “mongrel strength” of the wild, primitive human as Leto claims, Moneo is a fully domesticated purebred. He is a direct line descendent of the Atreides, and thus a result of 35 centuries of Leto’s behavioral and genetic design. In a way also similar to the purebred, he is high-strung. He spends the greater part of the book being really, really nervous any time he’s around Leto, for whom he serves as a right-hand man.
As Leto’s most senior employee, he knows as much about Leto as anyone can. He has read his history, sees his present and works to make his wishes reality. But for all that close association, he doesn’t appear to understand Leto very well; where Leto seems to respect strength and a level of rebellion and pushback, Moneo is consistently passive and cowardly. Where he serves Leto’s goals, the impression given to the reader is that he has been browbeaten into accepting them as correct.
As a possible result of this, Leto doesn’t appear to actually like Moneo; despite his usefulness, Leto spends most of his time talking to him in a way that resembles a bully verbally abusing someone he’s incredibly tired of. If he’s wrong in his actions, Leto is at least not wrong in his target; Moneo really does suck.
To the extent he cares about anything or shows even a scrap of bravery, it is in his desire to protect Siona.
If Duncan represents all that’s worth saving in humanity’s past, Siona represents humanity’s potential to transcend, and Nayla represents the balance of humanity’s nobility against its flaws, Moneo represents a sort of voluntary spiritual death in the face of large systems. His decisions are cool and mathematical, intended to maximize his performance within a system he dislikes but doesn’t dare to change.
To talk in detail about Leto’s goals, it’s necessary to talk generally about his fears. Those fears are for the most part robotic in nature. What alien life exists in the Herbert universe is broadly man-made, and the greatest historic risk humanity has faced was the crisis that drove the prohibitions of the Butlerian Jihad.
The Jihad is barely explained in the books, but seems to have involved a crisis of artificial intelligence; man rose up against the machine and won out, but the victory was either so costly or uncertain as to drive a universal ban on machines that imitate the complexity of human thought.
Leto’s ability to peer into the future does not tell him the specifics of the next danger that might threaten humanity, but he does know that it’s threatened, nonetheless:
“Nothing is ever separated from its source,” he said. “Seeing futures is a vision of a continuum in which all things take shape like bubbles forming beneath a waterfall. You see them and then they vanish into the stream. If the stream ends, it is as though the bubbles never were. That stream is my Golden Path and I saw it end."
Imagine Leto as a very big Big Yud (Eliezer Yudkowsky, rationalism’s original AI doom-sayer); he’s convinced that unless serious, committed action is taken the only future humanity can look forward to is paper-clip-based.
“The Ixians do not recognize that machine-makers always run the risk of becoming totally machine. This is ultimate sterility. Machines always fail… given time. And when these machines failed there would be nothing left, no life at all.”
Understanding Leto-as-Yud is necessary to understand the threat Leto is responding to, but is a sub-par way of understanding Leto himself. The better way is to look at Leto’s function as an all-powerful force both holding the hazards of malicious AI at bay while at the same working to mitigate those same hazards.
To put it another way, Leto might be organic and have a different distribution of strengths, but he functions as a sort of anti-AI AI, working a centuries-long gameplan to hedge against humanity’s impending doom. In the short term, his methods for doing this are pretty simple; he simply doesn’t allow people to build computers. His long-term plans on the other hand don’t involve shielding us so much as they involve teaching us to protect ourselves and making us capable of the same.
The first aspect of Leto’s work is to build a human that can threaten an AI in the same way we are threatened by AI today. Leto’s plans first involve developing something that mitigates his biggest advantage - that is, a strain of human that bypasses his clairvoyance. He finds this in Siona, who “fades from his sight”; if he can’t see her, then neither can other forms of oracle.
Though Leto spends the better part of 3500 years doing this, time is not the most vital aspect of his plan. The Bene Gesserit spent over 10,000 years on their breeding program, but focused on power alone; they succeeded in the form of Paul, but quickly lost control of their creation. Nor does the difference lie in creativity. The Teilaxu are short-term genetic engineers who exert a great deal of control in their alterations. In doing so they create wonders, but also bend humanity in unnatural and inhuman directions. They are universally hated, a sort of society-sized uncanny valley reviled by all. The Ixians put their faith in technology, speeding towards instead of away from humanity’s demise.
All three of these forces seek their own narrow goals while ignoring the broader needs of humanity. The Bene Gesserit got a messiah who immediately turned against them. The Teilaxu are mules who only vaguely resemble humans, and the Ixians are only kept from creating an immediate AI disaster through external controls exerted upon them.
Leto succeeds in helping humanity where others fail, not because of the time he spends and not even because of his resources; in the Dune universe, both could be duplicated by any sufficiently dedicated group of people. Leto creates no super-soldiers besides those that might have occurred naturally, given time. He creates no traits, instead only encouraging and curating traits that push humanity towards long-term viability. He keeps his power separate from his goals and thus keeps his goals pure.
The man-who-walks-as-a-worm thus protects humanity while actively encouraging it to grow. In doing so he creates something that will one day not need him, and that is also capable of destroying him. The book does not say so directly, but the fact that both of those are really one single goal is key.
An AI that was created accidentally may one day be destroyed accidentally, either by another AI or developing into an insane version of itself. Only by cultivating humanity itself into AI-destroying viability can it create safety.
Travel is restricted in Leto’s empire. While ornithopters and spacecraft still exist, he limits their uses entirely to official travel of the sort necessary to make the empire run. The vast majority of people are planet-bound and mostly restricted to such travel as they can accomplish on foot. They are trapped in place in a society he engineered to be banal and boring.
The goal (which is possible in Leto’s world, if not ours) is to encourage a kind of genetic yearning, a desire to travel and settle that is ingrained in humanity at a cellular level. Leto pushes for the accumulated voices of thousands of ancestors to scream “go west, my son” in unison, too loud to ignore. In the Dune universe, ancestral memories are a factual truth; this is how genetics work there. To build the desire, all you need is time - but the desire to do something and the wherewithal are two very different things.
Early in the novel, Leto makes an off-hand mention that some forces in the universe are terrified he will die away from water. When a sandworm dies, it breaks apart into sandtrout, which encapsulate environmental water until they’re surrounded by desert. Only when the environment is absolutely arid do they combine to create their adult sandworm forms.
The lifecycle of a sandworm is long, and TGEoD’s supply of spice is entirely contained within Leto’s horde; he carefully rations that supply to keep his empire functional. Leto expects that the remaining spice will be quickly exhausted after his death and for a period perhaps measured in centuries humanity will be trapped, unable to travel, starving, and creating harsh survival-of-the-fittest environments on all inhabited planets.
This great culling is intended to take the genetic advantage Leto has given and hone it to a razor’s edge. After a time, the re-emergence of the sandworm should reintroduce the spice to a population of untrackable super-warriors. Only by doing this does Leto believe he can scatter an ultra-tough humanity far enough through the universe that the machines will never catch them all, and that they will have a fighting chance even if they do.
“Yes, yes,” you are likely saying. “But what of the plot?”. And in asking you’ve stumbled onto the major problem with the novel: it doesn’t have one, really. Or at least not much of one.
Herbert’s books all have a theme. Dune is about teaching the reader about the untapped power of the desert, while Dune Messiah seeks to show them that power in action. Children of Dune is a warning that success can make you soft - since the harshness of the desert is what brings you power in this universe, bringing it under your control domesticates it and cuts into the very power base you rely on. When Paul Atreides blinks in the face of an overwhelming fate and is usurped by a younger, more vital generation, it’s no surprise.
TGEoD is only a slight departure from these themes, instead asking what would happen if the harshness and strength of the desert were concentrated in a single mind. Genetic development has always been a tale written by hardship, a story about how life rises to greatness only when facing and overcoming great problems.
(For the record, Heretics of Dune and Chapterhouse Dune are about sexing uppity women into submission and discovering a literal secret clan of space Jews, respectively. They are not necessarily required Dune reading, but if you are specifically looking for late-career-author-weirdness, they are excellent.)
Because TGEoD is mostly about describing the end-point of a millennia-long exploitation of desert power, it is better understood as a book of sci-fi political philosophy than a story. But, to the extent there is a story at all, it is as follows:
While Siona is busy running from wolves, Leto is killing one of his Duncan Idahos; the replacement Idaho is arguably the main character of both TGEoD and the rest of the series. The Idaho is brought in and shown the wonders of the future by Leto, and is momentarily convinced to be a faithful and committed servant of the God-Emperor. Despite Idaho having misgivings about some of Leto’s tactics, everything is momentarily fine.
This order is upset by Hwi Noree, who is immediately loved by Leto and who (again, weirdly) loves him back. After two short meetings, they decide to get married. But as you might expect, the kind of person who will get married after two coffee dates is not always the most reliable; after noticing that Duncan Idaho is sort of needy, she has one-time pity sex with him.
This does not result in a love triangle; Duncan now loves Hwi, but she explains to Leto that she acknowledges that he has Mua’Dibs on her and leaves Idaho in the lurch.
Meanwhile, Leto is busy kidnapping Siona, taking her to the desert and forcing her into a situation where she has to drink the spice essence he secretes to survive. Leto does this to force out her latent Atreides psychic powers, which shows her the golden path. In the past, this tactic has resulted in various people realizing the necessity of Leto’s actions and joining his team. Siona, however, is not convinced.
Duncan’s recently bruised ego makes him easy to recruit to her cause, and between her prescience and his rock-climbing skills (read the book, it’s weird) they manage to do the impossible, assassinating the god-emperor and shattering his 3500-year-rule. He passes over control of his spice-hoard to them, and they proceed to have a thousand children to seed the universe with a strain of human immune to prescient search.
Hwi Noree dies and, as a reader, you will find it nearly impossible to care.
So, why write a review on a 40-year-old sci-fi novel that nobody likes? If there is an orthodox strategy to winning an ACX book review contest, it’s probably something like “Review a non-fiction book about the connection between Tanzanite mining and the re-emergence of Georgist Monarchalism” and not so much “Scattered musings on the least popular Frank Herbert novel”. Why bother?
First, I think it’s an interesting example of prescience on Herbert’s part. Thirty years before you had heard of Bayes, Herbert had tapped into an assumption that seemed so clear to him he didn’t realize he should spend more time explaining it to you: that any development of AI or AI-like programs would logically (if not actually) eventually result in an AI that held views so profoundly counter to the needs of humanity that it would wipe us out in pursuit of them.
Today, those thoughts are old hat, but this book came out in 1981.
Second, there is always some value in anything that makes us think about the unpopular part of a problem. As is the case in any movement that addresses a potential hazard, eventually the prominent voices in the field are incentivized to focus on the sexy, sexy risk and not the boring, boring solutions. And thus you see much more “eventually nanomachines will fly through space at just under the speed of light, converting everything they see to AI”. They talk about the singularity, but less about potential solutions.
Let’s assume that both Herbert and Yud are correct and that an all-powerful AI is a given, that it’s coming like a freight train with little we can do to stop at least some form of it from becoming reality. If that’s the case, it’s arguably that the only possible solution that presents a potential “good ending” for humanity is not finding ways to avoid an all-powerful AI but instead moving as quickly as possible towards the correct version of the same.
TGEoD does not map perfectly onto reality. But to the extent it maps at all, it argues that the AI we build must be both protective and powerful; we have to allow it to accrue sufficient strength to prevent malicious AIs from gaining a foothold. We might be able to do that, or we might not. But if AI is inevitable then the dominance of a specific AI is likely; our best bet comes in promoting the correct one, not in the futility of trying to avoid the wrong program for the rest of eternity.
If we can attain a perfect protective AI, we can be happy while still acknowledging that we are at best its pets; the very concepts we are talking about demand that we acknowledge the AI in this hypothetical is superior to us. Therein lies the secondary risk of AI; by creating an AI that coddles us, we run the risk of never advancing as a race again. As Dune points out, soft environments tend to weaken rather than reinforce.
How much better would it then be if we could create an AI that restricts all other AI’s, but only as a secondary goal necessary to reach its primary objective of identifying what it is that makes humans human and amplifying those traits in a positive direction until we (as a still identifiably human race) can stand against AI’s on equal footing.
I am not an AI expert. I’m not sure if any of this is possible even without the theoretical framework of AI risk. But I do know people, at least a bit, and I know that the reflex to run from danger is often stronger than the impulse to run towards safety. If all-powerful AI is inevitable, then running away from the wrong AI is merely delaying our eventual fatal mistake. Running towards the correct AI is necessary. As Leto says:
Run Faster. History is a constant race between invention and catastrophe. Education helps, but it’s never enough. You must also run.
Footnote 1
Leto’s family, the Atreides(es?) are basically magic; the Bene Gesserit were essentially witches who could draw on the ancestral memories of women in their ancestral line, and they bred this ability into Paul, Leto’s father. As a bonus, they can also draw on male ancestor’s memories, as well as having a pretty good ability to predict the future and know what’s going on in the present.
If this seems like cheating, it is; Leto is not only 3500 years old, but also has the combined leadership/political experience of every member of his family line at least as far back as the ancient Greeks. Did I mention he’s also mentat and can do computer-level mental calculations? He can also do mind control with his voice, and roll into you to kill you like Sonic the Hedgehog. It’s like Frank Herbert was worried you’d mistake him for the Reasonably Tough Emperor of Dune and over-corrected in the other direction.