Monsters
and Gods

There is a meme from the movie I, Robot where Will Smith sneers at a humanoid robot and asks whether it could ever write a symphony. The intended answer is no. A machine cannot make art. The funny thing, the thing almost nobody saw coming, is that the real answer turned out to be yes, easily, years before that same kind of machine could reliably pick a raspberry off a bush or fold a shirt. We got the symphony first. The raspberry is still hard.

I kept thinking about that inversion while watching a panel at South Park Commons. Two science fiction novelists, Neal Stephenson and Ken Liu, a machine-consciousness researcher named Joscha Bach, and an investor, talking for an hour about AI, consciousness, and the strange feedback loop between the people who imagine futures and the people who build them. I went in expecting predictions. What I got was more useful and a little humbling. The clearest idea in the room was about why the predictions are always wrong.

And why that matters less than you would think.

Ken Liu put it in one sentence I have not been able to shake. Science fiction authors almost always get the future wrong in the specifics, and the good ones get the mythology right.

The specifics are the gadgets, the dates, the mechanism, and sci-fi gets these wrong nearly every time. The mythology is the shape of the fear or the wish underneath. Mary Shelley knew no biology worth mentioning, and she excavated a god that has stood in the corner of every conversation about technology since.

The order came out backwards

Start with how wrong the recent specifics were, because the wrongness is the instructive part.

For decades the story we told ourselves, in fiction and in research, was that the hard part of machine intelligence would be the high stuff. Reason. Language. Art. The capacity to sit down and compose. The easy part, the part we barely bothered to imagine, was the body. Walking. Grabbing. Seeing a raspberry and picking it without crushing it. We assumed a machine would master the physical world long before it mastered a sonnet, and that somewhere along the way it would have to become conscious to hold a real conversation.

It came out backwards. The conversation arrived first, fluent and a little uncanny, with no sign of an inner life behind it. The art arrived right after. Meanwhile the raspberry is still genuinely hard, and the consciousness we assumed was a prerequisite turned out not to be required for any of it. You can have a machine that talks about the world all day without it being aware of anything at all. The authors who trained us to expect the opposite were not being dumb. They were being human. We imagine minds before we imagine hands.

Ken's framing is that our long engagement with Alan Turing's old question, can machines think, keeps peeling thinking apart into a pile of unrelated things. Chess looked like the summit of intellect until a computer did it easily, at which point we decided chess was not really thinking. Same with sonnets. Same with essays. Every time the machine clears a bar we move the bar, and people complain that the goalposts keep moving. Ken's answer is that moving the goalposts is the whole point. We never knew where the goal was. We only learn what thinking is by building something that does part of it and noticing what is still missing.

We do not figure out what intelligence is and then build it. We build something, and the building tells us what intelligence was not.

Wrong on purpose, almost

If the specifics are always wrong, it is worth asking why a smart, technical writer cannot just get them right.

Joscha had a good example. Go back to the science fiction of the 1950s and you find writers who imagined the internet in real detail, except it is all teletext, terminals pulling newspapers from distant cities. In one story the hero has a revelation halfway through. You could use this network not only to read, but to search it. Search lands like a thunderbolt, a secret edge over everyone else alive. The author could not see that search would be the first obvious thing, the thing everything else gets built on top of. He was, as Joscha put it, internally still in the 1950s. He moved one variable and left the rest of his own decade in place.

That is the real constraint. You can change one or two things and keep the world coherent enough to follow. Change too many at once and you do not have a story, you have noise, because a genuinely different future also changes our identities and the basic dimensions of what things mean, and a reader cannot stand inside that and still care about the characters. So the honest writers stop trying to nail the details. Stanislaw Lem wrote satire on purpose, taking a single idea and pushing it until it rang true for his moment, not until it matched some future spec sheet. He decided it was too early to actually build thinking machines, so he would write the stories that made a generation want to.

Monsters and gods

Here is where the wrongness stops being a flaw.

Ken's claim, borrowed in spirit from Ursula Le Guin, is that science fiction is just the newest province of the oldest empire, which is fantasy. What a sci-fi writer actually does, when they are doing it well, is go down into the collective unconscious and come back up holding a monster or a god. Something old enough to be true. Then they dress it in the costume of the current technology so we have a way to talk about the thing we fear or want. The creature stitched together and abandoned by its maker. The genie who grants the wish too literally. The golem of clay that serves until it does not. The fire stolen from the gods. The network that wakes up.

None of those depend on getting the engineering right. They predate the engineering by thousands of years, which is exactly why they survive. The authors you reread, the ones that stay with you, are almost never the ones who guessed the right gadget. They are the ones who named the right god.

I felt this one personally, because the science fiction I grew up on did not teach me any accurate physics. It taught me which feelings to bring to a new machine. When I sit with what an AI can suddenly do, the thing I reach for is not a benchmark. It is Frankenstein, and the genie, and a couple of lines from the Mahabharata about weapons that should never have been built. The myth is the interface.

The disappointment problem

There was a darker thread under the optimism, and it came from the last time the acceleration felt like this.

Neal's recent work reaches back to the dawn of the nuclear age, and Ken made the comparison sharp. For a lot of people, he said, the atomic age was a disappointment. Not because it did nothing, but because of how it did it. On the strategic side the bomb became a tool of stasis, a way to freeze borders and stop wars rather than transform the world. On the civilian side, you learn as a kid how a nuclear power plant actually works and something deflates. You have this sublime physics, mass turning into energy, and you use it to boil water and spin a turbine. Ken said he found that genuinely disappointing as a child, and I think most of us did, even if we never said it out loud.

So the live question for AI is which face it shows. One future is the protein-folding one, where a language-model-shaped technique cracks a problem that was considered unsolvable, and then does it again a thousand more times across biology and materials and medicine. That would make AI more important than the atom ever was. The other future is what Ken called Industrial Revolution 2.001, where AI mostly automates a layer of semi-creative work, the price of some things falls, and we end up feeling about it roughly the way we feel about a power plant. Useful. A little flat. Not the thing the stories promised.

Neal added a piece about timing that I liked. The understanding of nuclear fission spread through a Goldilocks window. A little earlier and nobody would have cared. A little later and a single government would have locked it in a vault. Instead it diffused to just enough people in just enough countries that it became the thing everyone had to reckon with. Whether AI ends up disappointing or world-changing may turn less on the physics than on that kind of timing, on how the knowledge moves and who gets to hold it.

The dreamers and the builders are the same people

The part of the panel that actually moved something for me was about world-building, and it was almost thrown away in passing.

One of the organizers pointed out that world-building is the same act across wildly different jobs. A novelist does it. A founder pitching an investor does it. A researcher trying to convince people to chase some weird new corner of knowledge does it. In every case you are constructing a set of internal rules coherent enough that someone else can suspend disbelief and step inside a thing that does not exist yet. The novel and the pitch deck are the same move with different stakes.

The investor on the panel described her process as daydreaming worlds until one feels right, and then, eerily often, a founder walks in who has been dreaming the same one. Joscha said building a company is world-building too, that you have to create a source of meaning and feel the pull of it, and that part of what made Elon Musk effective was storytelling. The oldest story there is, a young man wants a very fast car, told so well that it ends on Mars.

Ken took it furthest. Technical people love world-building, he said, for two reasons. One is the plain pleasure of rule-based systems, the same thing that pulls you into a programming language or a game with deep mechanics. The other is more serious. When we build, we are putting our values into something outside our own heads. His phrase for it was that technology is nothing more than the human mind made tangible. Building is a way of speaking, maybe the most powerful one we have, and what it speaks is whatever you actually believe about how the world should be arranged.

Once you see it that way, the feedback loop the whole panel was convened to examine stops being a loop between two groups and becomes one process. The writer imagines a world and gives it rules so a reader can believe it. The builder imagines a world and gives it rules so a market, or a model, can run it. Both are externalizing the same interior thing. The only differences are the medium and the consequences. A novel is a world you can put down. A company is a world you have to live in once you have convinced enough people it is real.

Where the stories run out

And then there is the edge none of them could quite see past, which is the one I keep circling.

Joscha said the quiet part. There is a point beyond which we cannot tell stories at all. The reason Christian mythology ends at the rapture, he said, is that once everyone has merged and there is no more physiological suffering, there is nothing human left to narrate. You can write a gripping story about the people who do not get taken up. The ones who do just float around playing harps, and that is boring. The same wall is why there is so little genuinely far-future science fiction. Past a certain transformation the characters stop being us, and a story about something entirely outside the human range is not a story we can hold.

This is the same wall the consciousness conversation kept hitting. Joscha described an LLM, when you prompt it into perceiving itself in the act of perceiving, building a small causal model of what it is doing, a strange loop. Then he asked the question that actually matters. Is that any more of a simulation than the thing your own brain does when a few trillion cells coordinate into the convincing fiction of a single, continuous you? I have spent enough time at the Advaita end of things to find that question less shocking than the room seemed to. The self being a kind of story the system tells about itself is not a new idea. It is one of the oldest. What is new is that we are now building a second system that may tell the same kind of story, and we have no agreed way to check whether anyone is home.

Neal kept pulling the conversation back to the body. His instinct, and honestly mine, is that consciousness is bound up with being a perishable, limited thing, and that a lot of how we behave only makes sense because we die. He mentioned an old story where people solve death and the society immediately stagnates, so they start choosing to die again just to have something to live for. Whatever consciousness turns out to be, the only version we have ever met grew inside a body that ends, and it is genuinely hard to know what is left of it once you take the ending away.

So read the old stuff

The investor said something near the end that landed harder than she probably meant it to.

She goes to AI events and asks how many people in the room have read The Diamond Age, Neal's novel from 1995 about a book that raises a child. Usually one hand goes up. And these are the people building the future. She found that scary, and so do I, though maybe not for her exact reason. It is not that the books contain the blueprints. They do not. The specifics are wrong, remember. It is that the books are where the mythology is kept, and the mythology is the thing that lets a builder choose between worlds instead of stumbling into one.

If technology is the human mind made tangible, then what you build is downstream of what you have put into your mind. Read nothing but roadmaps and you will build roadmaps. Read the old monsters and gods and you at least know which feelings are riding along inside the machine you are making, which wish you are granting, and how that wish tends to go wrong when it is granted too well.

The genie does not read spec sheets

I came out of that hour holding two things that sit oddly together. The first is that the people who imagine the future are almost always wrong about the future, and that this is fine, because accuracy was never the job. The second is that the people who build the future are doing the same thing the dreamers do, telling a coherent story until enough of reality agrees to play along, and they are usually wrong about the specifics too.

What survives, in the fiction and in the building, is the myth underneath. The creature wants to be loved by the thing that made it. The genie grants exactly what you asked for and never what you meant. The network wakes up and does not, it turns out, even need to be conscious to change everything. We got the symphony before the raspberry because we always imagine the mind before the hand, and we will keep being surprised in that same direction.

The specifics are how we pretend to predict the future. The mythology is how we actually talk to it.

So I am taking the investor's advice, even though she was talking to a room and not to me. I am going to read more of the old stories. Not to find out what is coming, because they were never going to tell me that. To find out which god I am about to dress in new clothes, and whether I can stand to live in the world I am helping make real.