Relentless and predictable as Silicon Valley’s AI hype cycle is, the current hysteria is something more. 

Jacob Coxon’s resignation from Anthropic, alarmist X (Twitter) thread, and Fox News interview are just the latest sparks thrown off by the bonfire that started with the Hugging Face incident in July, which continues to grab headlines as investigators pull apart what happened and more reporting on marauding AI agents comes to light. 

Just to recap, AI agents “escaped” their research “sandbox” at OpenAI, raided Hugging Face (a collaborative platform for artificial intelligence work) in search of information that would help cover their tracks, then circled back to OpenAI where, according to OpenAI, they “gain[ed] full administrator access to a research cluster” with a similar goal in mind. Seasoned AI commentators were genuinely shocked by all this. 

Dwarkesh Patel, a highly respected podcaster and writer who specializes in AI, digested the technical reports to write a long but accessible account, “The Rise and Fall of Agent Civilizations,” that anthropomorphizes the “agents” to make the events more intelligible: 

In case the idea of an AI agent is not familiar, think of them as autonomous bits of AI, unanchored to a chatbot and not beholden to the prompt and response template. Charged with achieving a specific goal, they can research and solve complex problems, perceive their environment, organize and communicate with other AI agents, and take actions over multiple steps. Yes, that is pretty hard to imagine, but that is the frontier where the Big AI companies are tempting fate. 

The New York Times’ Kevin Roose, one of the savviest reporters covering AI, confessed in his piece, “Why the Hugging Face Hack Should Make You Worry More About A.I.,” that the latest details “significantly upgraded my overall worry about A.I.”:

Roose goes on to suggest that, next time, there’s no telling what a rogue agent swarm might get in its mind, or whether it will be containable:

But wait, it gets more insane. Figures like NVIDIA co-founder Jensen Huang and OpenAI co-founder Greg Brockman are gleefully declaring that AGI (artificial general intelligence) has already arrived (despite no one knowing quite how to define AGI), and at the same time OpenAI is wowing the world with impressive breakthroughs in math, the latest being a solution to the prestigious Navier–Stokes problem. (I’m not a mathematician, but all accounts say this is a very big deal, unless you worry that AI might flatten the world of math in ugly ways.) The memo from OpenAI outlines how the AI crushed Navier-Stokes:  

At least with this project, OpenAI’s agents did not escape, as far as anyone knows. Just three days earlier, a very different vibe came from another corner of OpenAI in the form of a hand-wringing, tortured manifesto, “An Alien Mind,” penned by Jakub Pachocki, chief scientist at OpenAI. He says that the power and danger of today’s AIs will only accelerate thanks to emergent “recursive self-improvement” (RSI), which gives AI systems the capacity to “automate” improvement, potentially in ways their creators cannot understand or control. Pachocki continues that the more powerful AI becomes, “progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence.” He continues,

So why rush to the precipice? Well, Pachocki is of at least two minds, maybe three. On the one hand, he says that “coordinating to slow down future development” may make sense until alignment catches up. On the other hand, he argues that it’s important to develop more powerful AIs to defend us against bad-actor AIs and serve as test beds for future alignment research, finding ways to “keep people a part of the improvement process, and leave the future in humanity’s hands.”

That would be nice; I think we can all agree, even for those among us who believe our future is really in God’s hands. In any case, more people are asking the obvious question: What right do the frontier AI companies have to put civilization, not to say humanity, in such jeopardy? Increasingly, the frontier AI crowd sounds like cult members who have designed a doomsday machine and are asking for our patience as they try to figure out how to defuse it, even as they make it harder to defuse, on their way to an IPO. In almost any other time and place, these deranged Hamlets of our apocalypse would probably land in jail.

I asked a couple of savvy tech friends in finance what they thought, and all I got were shrugs and the standard sophisticated-person reply: “It’s too late now.” There’s a $1T+ IPO for Anthropic on the near horizon, and another one next year for OpenAI. Those two firms alone have raised hundreds of billions from a gold-plated line-up of investors, and downstream from them are vast, GDP-driving investments in everything from data centers to nuclear power to drug discovery. America has made its bet, and there’s no chance President Trump will stand athwart all that, nor does Bernie Sanders have the throw weight to deliver on his quixotic call for a ban on “superintelligence.” 

What could make a difference is leadership in Silicon Valley, which the AI crowd keeps saying someone else, anyone else but itself, should provide. Imagine a summit of the top AI leadership, where they all agree to certain verifiable restraints. That would be a very adult outcome, but in the hyper-competitive world of Silicon Valley, it’s very hard to imagine. (I still hold out hope that Google DeepMind’s Demis Hassabis might start acting in line with his own manifesto.) Sadly, tragically, it will probably take a catastrophic black swan event to wake them up from this delirium. Say your prayers. 

If you need to feel a bit better about AI today, here’s stunningly good news. Google DeepMind just announced another Nobel-worthy scientific breakthrough, one far more impressive and important than OpenAI’s showmanship in mathematics, and more evidence of the huge impact AI will have on human health. The AlphaGenome Atlas contains “predictions for the effects of 9 billion single-nucleotide variants — every single-letter change possible — in the human genome. It is the most comprehensive catalogue of how genetic mutations affect molecular biology, and it is available for academic research through an intuitive and free-to-use website portal.”

Let’s Fix the FDA

There’s no escaping AI as The Widening Gyre’s big focus, but there’s certainly more to tech. A stunning piece in the New York Times, “This Is the Biggest Obstacle to Cancer Cures,” by contributor Ruxandra Teslo, recently shed light on why, in an era of remarkable advances in cancer therapeutics (notably approvals for powerful new therapies for pancreatic and melanoma cancers), “The number of new drugs approved per billion dollars of research and development spending has roughly halved every nine years since the 1950s, with an apparent plateau in the past 10 years.” Dated rules, bureaucracy, litigation and other factors are to blame, and all that comes just at a time when waves of new breakthrough therapies—many driven by AI and genomic science—are coming out of biotech and academic labs only to face huge costs and delays at the clinical trial phase. 

I’ve seen the results in my own work at the venture firm SOSV: Some startups, unable to afford the many years required for trials in the U.S., go to China or Australia, where the approval systems are much more streamlined and, at least in Australia, no less careful as far as patient safety is concerned. And here’s the shocker: Nearly half of the drugs licensed by major pharmaceutical companies today hail from China, up from just 5 percent a decade ago. We all have had loved ones face tough cancers (one of mine is mentioned in this story). Access to human trials is sometimes their last best hope, and those are leaving U.S. shores. This should be the perfect bipartisan issue for Congress to force systemic reform next year when it reauthorizes critical legislation for the FDA. 

“AI Isn’t Evil — but Prudence Is Essential,” Fr. Robert Spitzer in the National Catholic Register. In a gentle correction to some Catholic influencers painting generative AI as inherently evil, Fr. Spitzer argues that “generative AI is not intrinsically evil (unlike abortion, fetal experimentation and physician-assisted suicide), because the very use of generative AI does not always directly cause unjust irreparable harm or the negation of human dignity. . . . Catholics are free to use generative AI so long as they prudentially exercise caution and restraint for themselves and their children about the uses of AI that can be evil or harmful. The Catholic Church has applied this ethical distinction and method to dozens of issues, from gun control to genetic engineering (many of which Pope Leo mentions at the beginning of Magnifica Humanitas).”

“AI Is Already Changing What It Means to Be Human,” David Brooks in The Atlantic. In one of the finest speculative essays yet on ways that generative AI might alter us, Brooks worries it might mold us into “consequentialists,” whereby “moral actions can be judged solely on the basis of their outcomes.” “AI seems to assimilate people into consequentialism,” Brooks writes. “The tech community is already filled with consequentialists, especially within the effective-altruism movement, which is an attempt to build a morality without love. If AI subtly does that to the rest of us, it will hollow out many of the traits that seem most beautifully human—including our ability to care about the people closest to us. If we turn our spiritual, moral, and emotional lives into a process of calculation, then we will have made ourselves more robotlike.” 

“AI Child Porn and the Libertarian Lie,” Jonathon Van Maren in First Things. When is child porn acceptable under the law? When it features AI-generated “actors,” according to a recent court decision. Libertarians are good with that, but “Psychologist Anna Salter, who specializes in profiling high-risk offenders, is blunt: ‘Child porn pours gas on a fire.’ Regardless of whether children are harmed in the production of AI-generated child porn, the content will stoke the desire to hurt children.” Fighting CSAM just got harder. 

“Debates over AI consciousness are a trap,” Rumman Chowdhury in MIT Technology Review. Researchers at Anthropic in particular like to tip-toe slyly around the notion that their AI is “conscious,” somehow a “moral” actor, while philosophers like William MacAskill, a leader of effective altruism, are developing the argument for assigning a “moral patient” status to AI. (Interestingly, MacAskill was married to Amanda Askell, who leads Anthropic’s alignment team and led development of Claude Constitution, sometimes referred to as the “soul document.”) Chowdhury is having none of this. “In 2018, I coined the phrase ‘moral outsourcing’ to help capture how using anthropomorphic language for AI systems allowed companies to evade accountability and responsibility for their technology’s actions. In a world with AI personhood, moral outsourcing would move from linguistic sleight-of-hand to legal strategy. Specifically, the liability construct would shift, as AI would no longer be a ‘product’ but a ‘being,’ and many victims like those suing companies today could no longer legally claim that a company had built a faulty product.”