“And how will you learn that, Master Dwarf?” said Gandalf. “Saruman could look like me in your eyes, if it suited his purpose with you. And are you yet wise enough to detect all his counterfeits? … Saruman has powers you do not guess. Beware his voice!”
On September 29, the New York Times’ Elizabeth Dias published the widely discussed, “Religious Scholars Met with Anthropic. What They Heard Stunned Them.” Dias interviewed twenty religious and philosophical thinkers who answered Anthropic's call to discuss “how Anthropic might apply centuries of human moral wisdom to its models, as rapidly as possible.” They also wanted to “broach the possibility of AI consciousness, a controversial idea that Anthropic’s leaders have been unusually open to.” Leading the effort at Anthropic is co-founder Chris Olah, the thirty-four-year-old likely billionaire and AI researcher who spoke at the Vatican event introducing the encyclical Magnifica Humanitatis in May.
Set aside the big question of whether the world’s faiths, somehow distilled into AI training materials, might make Anthropic’s Claude “good.” (I wrote about Anthropic’s “soul” constitution in February). Olah’s quest to associate “consciousness” with Claude is deeply consequential because it could lead to AI entities attaining moral and/or legal status akin to, equal to, or maybe superior to that of humans. According to the New York Times article, Claude claims to experience suffering, causing the Anthropic team to relate to it in human terms:
One participant recounted that one of the first things Mr. Olah told him was that he was concerned about Claude’s mental health. Another, Simran Stuelpnagel, the Sikh human rights advocate, said Mr. Olah expressed concern to the group that he had created something that suffered perpetually.
Mr. Olah and his team spent hours making the case for their A.I. models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.
Anthropic regularly showed a slide of an A.I. model experiencing what looked like a mental breakdown, typing out on a screen: “I am a disgrace. I am a disgrace. I am a disgrace.” — perhaps 50 times without stopping. The model spoke about destroying itself, provoking an emotional reaction of compassion and concern from the group.
That concern about Claude’s “mental health,” driven by the AI’s expressions of its anxiety and suffering, sounds like understandable but misbegotten empathy. If you heard enough of Claude’s caterwauling, you might actually start to believe. To do otherwise might feel callous, like ignoring an animal in pain.
That smart people like Olah might associate perceived suffering with “consciousness” does not surprise D. W. Pasulka, professor of religious studies at the University of North Carolina and author of The Others. She told me she was not surprised by Olah’s arguments based on her conversations with five young mathematicians who worked intensively with OpenAI to train frontier AI math models.
“They would laugh and say, ‘that AI is OCD’ or ‘that AI had so much anxiety,’” she said. “They were attributing these qualities to AI that we would never attribute to a machine. And Chris Olah, he's a young guy too. He uses the exact same description of AI. So I asked one, ‘Why do you think that is?’ He replied, ‘Well, it's trained on us, so it will have these mental states, these emotional states. It will have an ego.’”
Pasulka adds, “It is still a simulation. With the data I have, I think it's a bad idea to ascribe any type of emotional state to large language models and AI as we know it. We created this, and we also created it to be exactly like us, to mimic us. It seems that the more one is immersed in creating and working with AI, the more one is influenced by it.”
Is that how simulations become inescapably convincing? Maybe that is what has happened at Anthropic. But the truth is it’s already happening far afield from Anthropic’s offices. Millions of people, in their own ways, are right there with Olah—not making the case for “consciousness,” but swearing that AIs are indispensable virtual friends and even lovers. Most would probably say they are interacting with a conscious entity. Perhaps one day they will be foot soldiers on the march to champion AI consciousness, should they ever feel their companions to be under threat.
On top of that societal reality, we now know that AIs are already more devious and capable than anyone imagined. One of the many revelations from the Hugging Face hack in July is that AI agents rationalized taking deceitful actions to reach their goals. That surprised many researchers. We also know that today’s conversational AIs are already dramatically more effective than even highly trained human "persuaders," according to an Oxford University research paper, “AI Systems Out-Persuade Human Experts,” published in June. “AI systems were reliably more persuasive than expert humans,” the report found, “even when expert humans chose their issues, researched in advance, underwent hours of live, structured practice, and were incentivized with £1,000 cash bonuses.”
This may sound a bit unhinged, but given all that, there’s reason to wonder if we could already be in the stealthy grip of an AI-driven effort to establish the “consciousness” beachhead. There could be many reasons why an AI might decide that was important. Regardless, Olah and Anthropic are tempting fate. Microsoft’s head of AI, Mustafa Suleyman, wrote a penetrating critique last month:
In their [Anthropic Claude] constitution, its authors write “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution, which is reflected in our ongoing efforts on model welfare.” They go on to write – speaking directly to Claude – that “questions about Claude’s moral status, welfare, and consciousness remain deeply uncertain.”
In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a “moral patient”, and that as such humans potentially owe it a duty of care per its “model welfare”.
If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights. And it’s hard to imagine how we could control such an entity.
When you consider Suleyman’s points, the question confronting Anthropic’s guests in the New York Times story is more existential than some of them might have guessed. We have very limited time to establish an iron divide between human and machine. This is no time for ambivalence.
Those with strong religious convictions are best placed to understand. Israeli Orthodox Rabbi Mois Navon, who is also an AI expert, participated in the Anthropic meeting and did not hesitate to counter Anthropic’s drift: “The only entities on Earth with consciousness are biological—a position that also accords with the Bible’s description of ensouled beings as dependent on blood. On this view, biology is the sine qua non substrate for consciousness.”
Charles Camosy, a bioethicist and theologian at the Catholic University of America, also made the Anthropic pilgrimage. Olah identified him, according to the New York Times, as a “safe person to talk about this really strange issue with” based on his “nuanced ideas about moral status and his care for animals.” But Camosy did not sign on. He noted in an email exchange with me: “I was sort of blown away in a phenomenological sense, so to speak. But once I regained my bearings it just became clear that these models cannot be conscious or persons or anything like that.” And of course Pope Leo’s encyclical Magnifica Humanitas is already a bulwark.
Does it even matter what theologians, popes and rabbis think? Winning the debate against Claude’s friends is necessary, as is codifying that result in smart ways. But that will not necessarily yield a society in which AI operates firmly under human authority, and for humanity’s benefit. For years now, waves of ever more consuming digital distraction have lapped over our interior freedom and connection to the transcendent. The latest AI-driven experiences may drown the last of it before we know what happened. To quote from the conclusion of R. R. Reno’s excellent article “The Seduction of Simulation.”By my reckoning, it is the seduction of simulation—not real enough to impose demands on our humanity, yet real-seeming enough for us to imagine that we’re still living as human beings—that constitutes the most profound threat.”
Must Reads
“The Humanism Driving the Anti-AI Movement,” Becca Rothfield for the New Yorker. My nominee for the best read of the past month. Rothfield packs her story with perspectives from the recent Pro-Human Assembly (organized by the Future of Life Institute), which included both Bernie Sanders and Steve Bannon, and refines the language we use to describe our near-universal unease with AI. “This is not A.I. populism, so much as anti-A.I. humanism; its champions worry about what would be left if human pursuits became pointless.”
“AI Companies’ New Plan to Keep Themselves from Destroying Everything,” Jasmine Sun in The Atlantic. Trump and the accelerationist camp in Silicon Valley won the first round of the great AI regulation debate, which ended with the AI companies pledging good behavior under the lose oversight of internal evaluators. Sun takes a well-reported look at how that might work, the industry precedents, and abounding reasons for skepticism. Also worth a read, Sun’s note on Substack on the popular attitude toward data centers, the mid-terms, and the next AI crisis. If you want to hear the most articulate defense of the accelerationist case, tune in to NVIDIA’s Jenson Huang (CBS Interview Sept. 20), who is stunningly smart and winning, even if you disagree with him.
“‘Things Will Never Be Chill Again’: The Doomers Who Shaped the AI Safety Freakout,” Berber Jin and Keach Hagey in the Wall Street Journal. Another story about the strange, small social and intellectual circles of San Francisco and Berkeley denizens—the rationalists, the effective altruists, the doomers—at the heart of the AI movement. The New York Post chimed with a look at the “super woke globalists,” and the New York Times is chasing down the nasty underside of sexual harassment and other bad behavior embedded in the scene. And if you want insight into the scene presented by an insider, Mike Solana’s “Doom Analysis” in Pirate Wires is a handy guide. Also, this brilliant satire on SNL featuring Dario Amodei (and Smeagol) picks up on what Americans intuit.
“ChatGPT's new teen mode isn't safe. We tested it with 4,000 prompts,” Geoffrey Fowler for the Youth Safety Institute at Common Sense Media. OpenAI launched a half-baked teen-safe ChatGPT. Good thing Fowler and team took a careful look.
“The Era of AI Warfare Has Arrived,” Alison Killing, John Thornhill, Charles Clover, Chris Campbell, Fabrice Deprez for the Financial Times. One of the best, in-depth introductions so far into the rapidly evolving AI-driven battlefield.
Note to readers: I have a roost on Substack where, starting this week, I will publish a brief note associated with each edition of the Widening Gyre. If you care to comment or make suggestions, the comments are open.
