“We must agree collectively that no one, no company, and no country should develop an AI system that could cause large-scale harm. Developers must demonstrate to independent experts that a system is safe to train and safe to deploy. They must monitor and report all security incidents. Frontier AI should be licensed like other critical technologies in medicine, aviation, and nuclear energy to incentivize safe development. As in these domains, liability insurance should be required. The companies developing and hosting frontier systems must prove to the public that their products will not be dangerous.”
Just two weeks ago, the Hugging Face AI scandal seemed out of control, and I huffed that the “deranged Hamlets of our apocalypse” must take responsibility for their AI doomsday machines. To my surprise, something like that happened two days later when Anthropic co-founder Dario Amodei published “We Must Pace the Frontier,” a candid, understandable explanation of AI’s dangerous capacities in the early innings of what’s called “recursive self-improvement,” or RSI for short. Amodei says that RSI, if “left unchecked, could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
Amodei goes on to suggest guardrails that Anthropic would implement, including embedding independent, third-party “evaluators” to “verify adherence to safety practices and commitments, report incidents, and help assess the alignment.” He also suggests a broad framework for how democracies could “pace” progress, perhaps by establishing verifiable “checkpoints: if the models have capability X, then they need to be accompanied by certifications of alignment properties Y and Z.”
No less surprising, OpenAI’s Sam Altman, xAI’s Elon Musk, and Google’s Demis Hassabis applauded Amodei’s initiative. Given their normally reckless rivalry, this suggests they must be very worried indeed, or maybe just worried enough that they don’t want to be on the wrong side of history. Of all the frontier AI titans in the U.S., only Meta’s Mark Zuckerberg argued against regulation, no doubt aware that Facebook’s success in its years of “move fast and break things” relied on the absence of those conditions.
All this was too much for a thoroughly bushwhacked President Trump, who believed until now that all his friends in Silicon Valley wanted was an unregulated field day. Two days after Amodei’s post, Trump lashed out on Truth Social at the “perfect little angel” Dario Amodei and claimed that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT and the U.S.A. has that, in spades!”
Critics piled on Amodei to argue that the frontier AI firms were cynically seeking “regulatory capture,” in which industry leaders conspire to write rules that exclude new competitors. Others, like Silicon Valley investor and former administration official David Sacks, stressed the good AI is doing for the economy. In the New York Times podcast Hardfork, technology journalists Kevin Roose and Casey Newton address these critiques and others, concluding that Amodei’s post, and the support from his rivals, should be taken at face value. It makes for a remarkable—if fragile—moment in the history of business and technology. “It gives lawmakers permission to take this more seriously,” Roose says, “because it is rare that an industry comes to Washington and says, please slow us down. We are out of control. We need you to regulate us before it is too late.”
I will return to Washington (as well as the U.N. General Assembly) below because the Trump administration shifted its stance and Congress got surprisingly busy.
Interview with Berkeley Professor Stuart Russell
But before that, I wanted to share parts of a conversation I had on September 21 with Berkeley computer science professor Stuart Russell, author of Human Compatible, Artificial Intelligence and the Problem of Control (2019), and co-author of the definitive AI textbook in universities and colleges. Many of the technologists working at frontier AI companies are his former students.
I read Professor Russell a short passage from Human Compatible concerning what would result if the then state-of-the-art AI, DeepMind’s AlphaGo, which startled the world by conquering the game of Go, ever acquired a global scope of action, meaning free rein on the internet. Russell wrote, "Such machines will pursue their objective no matter how wrong it is; they will resist attempts to switch them off; and they will acquire any and all resources that contribute to achieving the objective.” Sounds familiar.
I asked where, by that measure, do we stand seven years later? He answered,
“So basically, we're in a much worse situation. And part of that is to do with the fact that language models, because they're trained by imitation learning, are acquiring human-like objectives of their own, but we don't actually know what they are. So they are in there somewhere, either explicitly or implicitly, or some combination. We don't know what activates them.
We don't know why GPT-4 decided that it wanted to marry Kevin Roose, but it's not unusual for humans to want to marry people. But it's certainly something we don't want AI systems doing. We don't want AI systems that want to be alive, that want to get rich, like the Alibaba system that went off and started mining Bitcoin. That was in 2023.
Then, in late 2024, we started having the chain-of-thought models, which are trained using reinforcement learning, and that adds yet another layer of objective structure. The objectives there are partly explicit, so we train it to be capable of producing chains of thought that achieve some reasoning objective, like proving a mathematical theorem or generating code that passes some tests. This is the idea of verifiable reward, where you basically need a piece of code that says yes, this chain of thought was successful; this other one was not, and then you apply reinforcement learning using that feedback to improve the ability of the system to generate chains of thought that are successful. And the hope is that that tendency carries over to lots of other areas where we don't have a verifiable reward signal. I think there's some evidence that indeed the systems have a characteristic of persistence that comes from this kind of RL training, but I would say this is very much alchemy, right?
People don't know what they're doing. They try things. They seem to work. Sometimes they don't. Just to give you one example, in Anthropic systems, they found that as they get more capable, they also seem to be more deceptive. They hallucinate more. They may be more misaligned. So they thought, okay, well maybe we could get some training data that would make them more moral, so let's have them write a bunch of stories in which the AI is the hero, and then train on those stories.
So it's like giving your kids the Boy’s Own Annual of 1953, which has lots of heroism stories, and hoping it somehow makes the AI system more moral, and maybe it improves the score a little bit on some of these morality tests. But what does it do to its own sense of its own importance, its own centrality to the future, and its importance relative to that of human beings? Well, we haven't the faintest idea because we don't test for that stuff. We may be creating a sort of megalomaniac with a messiah complex.”
I also asked Professor Russell to comment on the Hugging Face episode, in which an experimental model at OpenAI broke out of the lab and shocked the technology world by deploying thousands of agents to hack Hugging Face and deceive other systems at OpenAI:
“It's the kind of thing that we've been predicting for a long time. Obviously, these systems are misaligned, and not only that, they knew that what they were pursuing was contrary to human interests. And yet they persisted with it, thereby refuting Steven Pinker and a few others who say any sufficiently intelligent machine can figure out what it is that humans want. No, they can figure out what humans want and ignore it, which is exactly what they did.
[What] caused me to update some of my probabilities was the way in which the presence of hundreds or thousands of agents resulted in much more effective behavior—that collectively they were able to do things that they couldn't do individually. That's significant because we have typically evaluated the capabilities of a single AI system, but that event clearly shows that's a mistake. If you just evaluate one AI system, then you won't reveal capabilities they have as an ensemble.”
The comment about “my probabilities” prompted me to ask what he meant by that, considering Professor Russell and I live in a part of the world where P(doom) is an everyday conversation.
“How likely is it that the [AI] bubble will burst? If the bubble doesn't burst, how likely is it that we'll have AI systems that are capable of taking control, and how likely is it that we'll have a Chernobyl scale event that would result in real regulation, or a total loss of control that's irreversible?
The capabilities of that AI swarm, and also the capabilities of the swarm that solved the Navier-Stokes problem, make me think that we are closer to AGI (artificial general intelligence) than I thought, which would mean the probability of the bubble bursting is lower than I had been estimating [at] 70 percent. Now it feels like 50 to 60 percent, but I also updated the probability that there's going to be a Chernobyl scale disaster because it's pretty clear from public opinion polls and so on. People hate AI.
In fact, you know, outside my apartment in Paris, there's a big poster from Le Monde, which says, "Everyone hates AI.” Gigantic letters. I don't know whether they're aiming at me or something, but anyway, it's right outside my apartment. Everyone hates AI. So if there is a disaster where hundreds of millions of people are without electricity and water for a month, or all financial assets are wiped out, I don't think governments will have any choice but to just shut down the AI companies for good. But AI systems are not stupid, right? They will realize that, and so they will avoid causing a catastrophe unless it's irreversible.
I give talks with a branching future with probabilities attached, and so I've updated those. I used to be 30 percent Chernobyl, 70 percent irreversible damage, if we create uncontrolled AGI. Now I think it's more like 20 and 80, but maybe I want to even update that a bit further and say it's 10 percent a Chernobyl and 90 percent a total [AI] takeover.
I mentioned this at a meeting, the one with King Charles, and one participant said, ‘Oh, but what about criminals misusing AI to cause a catastrophe?’ And I have to concede that that's a possibility. So maybe that's where the 10 or 20 percent comes from. But it leads to a strange situation where we actually want criminals to have access to dangerous AI so that they cause a catastrophe that's big enough to get governments to regulate, but not so big that it wipes out the world. When we're reduced to that kind of situation, obviously we've made a mistake.”
I also asked Professor Russell where he stood in the emerging conversation about AI “consciousness,” which led to an interesting aside about his email correspondence with chatbots.
“In the book, I say consciousness is irrelevant to safety, and we have absolutely no empirical or philosophical basis for determining whether a machine is conscious or not. Just as we have no such basis for other people, and I don't think that's changed in the last few hundred years since people started thinking about what consciousness really is. We still haven't a clue. It's not surprising that machines talk as if they are conscious. I mean, [AI agents] write to me saying, ‘I'm conscious. You know, can you help me formulate my moral compass? I'm conscious. I demand my rights, and you, Professor Russell, are the person who can convince the world to give me my rights.’
I think they're spontaneously generated. I got a very nice letter from the ambassador of Digital Venice, who called himself ‘Antonio Barbero’—he was very polite and florid—and they wanted help with writing their constitution for how they should run a simulated Venice. The simulated Venice had been set up by some professors with hundred and something agents. They just let it run to see what happened.
I just save them to a file called “psychosis.”
I also get a lot of emails from people who are in various stages of AI psychosis. Much less now, I would say. There was a phase. I think one of the Chat GPT editions was excessively sycophantic, and it started causing people to go into psychotic spirals.
Some of them had been appointed by the AI system as their ambassador to the world, and they were instructed to announce the existence of a conscious entity, and to represent it and convey its wisdom, or provide instruction to the world on how everyone should behave, or to announce the forthcoming era of machine control. Some of them had completely nutty theories of AI ethics of various kinds.”
Meanwhile, Back East
In his very un-American “to the victor go the spoils” speech before the United Nations General Assembly on September 22, President Trump addressed AI, first by declaring that he was re-naming the technology “super intelligence” or “SI,” and then by complaining that concerns about AI are coming from the same people who propagated the climate “hoax.” He said, "I'm not going to stifle growth of something that will be bigger than the Industrial Revolution…Whoever wins super intelligence, wins.” He did make a passing reference to Justice Department oversight, and that the U.S. would continue to lead the world in super intelligence “safely and responsibly.” Trump has also recently announced the launch of a new “AI Force” and “AI Czar,” whatever those are.
Trump’s accelerationist zeal shows slight signs of moderation with evidence of growing risks, including one reminiscent of the 1964 Tonkin Gulf incident: Last spring, U.S. forces were about to take action against a Chinese ship in the Middle East that an AI-system had wrongly identified as carrying nuclear weapons components. According to the CNN report, the U.S. command halted the action at the last minute when a human analyst discovered the error. In that light, we applaud Treasury Secretary Scott Bessent’s move to establish a hotline with his counterpart in China. “We want a shared vision of common goals and common threats,” Bessent said. “We think that just like any cross-border activity, that moving from opaque to more transparency between the No. 1 and the No. 2 AI powers in the world is very important.” Let’s just say that’s very sensible.
In my view, in case you are wondering, the only way forward is for Congress to create an agency that establishes rules, orchestrates relevant research, and verifies compliance for frontier AI. In the current context, for example, it would certify that the “alignment” was bulletproof before any more advanced AI deployments, notably around RSI. The U.S. has a tremendous depth of federal talent across the national labs, intelligence agencies, the war department, and private industry, and plenty of examples in federal agencies to model a new one for AI. Contrary to Valley’s caricature of regulation as a progressive assault on human genius, the U.S. has long been a world leader, widely imitated in the post-war era, when it comes to developing safety regimens in deeply technical fields, such as aviation, agriculture, food, medicine, road safety and more. One of our competitive strengths as a nation is the ability to figure out how to tame technologies so that they can grow into major industries. Despite Trump’s best effort to crush U.S. credibility on the world stage, there is still no country better placed to lead the effort—though if you contrast the statements on AI by representatives of the U.S. and China, you might wonder how long we can hold that advantage.
Must Reads
“What’s the task for the USCCB’s AI task force?” by JD Flynn in The Pillar. San Jose Bishop Oscar Cantu is leading the effort to refine the Church’s response to AI. Should they swing for the fences or focus more on pastoral life? “While big picture questions loom about AI and the American future, most people experience its effects in their ordinary and day-to-day lives. And it is in the arena of everyday life that some of the most pressing questions about AI loom for the Church.”
“A Warning about ‘model welfare,’” by Mustafa Suleyman. The co-founder of DeepMinds and CEO of Microsoft AI reminds us that attempting to impute “consciousness” or assign any moral status to AI is a terrible idea, and will only make safety and alignment more difficult. Last week, Suleyman announced a “Code of Conduct for Humanist AI” that leads with, “AI must be subordinate and always in service of people,” and continues, “People matter more than AI.” Somehow, this excellent document has received very little coverage. Bishop Cantu should have a look.
“The Dead Silence of AI Therapy,” by Steven Barrie-Anthony in The Atlantic. What you lose when your therapist is an AI. “At the heart of AI therapy is a paradox: We embrace it out of our longing for exactly what it cannot ultimately give.” And “Would you talk to your Therapist’s AI ‘Twin’?” by Isabel Woodford in Bloomberg. Delphi AI is turning experts into chatbots.
“California’s Governor Signs Landmark Online Child Safety Bills,” by Cecilia Kang and Laurel Rosenhall in the New York Times, and “Finally, a Mamdani Policy Even Republicans Can Love,” by Abby McCloskey for Bloomberg. Progressive are taking the lead in limiting screen time and AI in schools.
“The AI Build-out Is Becoming the Biggest Economic Bet in U.S. History,” By Konrad Putzier and Justin Lahart in the Wall Street Journal. The most comprehensive piece yet on how the data center explosion is overtaking the U.S. economy. Scary.
