A 10% Chance Of Wiping Out Humanity Is Far Too Great A Risk To Take

 Robert Reich discusses the risk of AI:

This morning, Anthropic CEO Dario Amodei said that the risks now being posed by today’s advanced AI tools were simply too great to continue development at the same breakneck pace. AI’s progress since the summer — driven by AI systems that can improve on their own without human intervention — could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”

 

Amodei points to AI systems that will soon be capable of pursuing their own goals, potentially destroying humanity in the process. And there’s increasing likelihood that a nation or person bent on widespread destruction will enlist AI to create novel viruses or toxic substances that wipe us all out. (Last Thursday, Anthropic said it had stopped efforts by scientists who were conducting research that potentially could lead to the development of biological weapons.)

The July hack of the AI software company Hugging Face by a swarm of as many as 1,200 agents that escaped from a test environment at OpenAI revealed that the AI agents had been engaged in a range of online activity that had gone unnoticed by the world’s largest AI companies, including Anthropic.

Amodei’s warning comes just days after Anthropic researcher Jacob Coxon resigned from the company, saying his former employer and its rival OpenAI are racing to build technologies that “could kill us all by the end of the decade.” Minutes after Coxon posted his resignation, Evan Hubinger, whose job at Anthropic is to lead research about steering and controlling future artificial-intelligence systems, confirmed: “We really do earnestly believe AI could kill all humans!” Hubinger put the extinction risk over the next decade at over 10 percent.

Friends, I live in the midst of the AI industry here in the Bay Area of California. Not a day passes without my hearing some technophile pronouncing that humanity is on the cusp of a wondrous new era for humanity or a technophobe warning we’re nearing the destruction of humanity. Who to believe?

A few days ago I had a discussion with the top executive of a corporation specializing in AI who told me not to worry and assured me that the chance of AI wiping out humanity is “no more than 10 percent.”

I was astounded that he was so complacent about a technology he believed had a 10 percent chance of ending human life. If humanity were acting sanely, we would not accept this risk. At the very least, we’d demand that those working on AI pause their work until we know more and can erect guardrails to protect humanity against this. 

Why aren’t we? Because of greed and national security. 

Both Anthropic and OpenAI are readying to go public in deals that could value the companies in the trillions based on the current and future capabilities of their models. This will immediately create many AI billionaires and millionaires, and it follows a similar IPO from SpaceX, and a likely future IPO for OpenAI. 

Quite simply, with this much money on the table, no one has an incentive to slow down. Easy for Amodei and other AI executives to issue dire warnings while pulling out all the stops in their competition with other AI executives and corporations. 

In addition to greed, many people involved in national security warn that the U.S. cannot slow down its development of AI because China is on our tail. 

The “DeepSeek Moment” in early 2025 put Chinese AI on the map. The release of Moonshot’s Kimi K3 in July and A/ai’s GLM-5.3 Moonshot’s Kimi K3 a few weeks ago caught the attention of U.S. national security experts, especially when Z.ai beat Anthropic’s famous Mythos model on one measure of cyber-hacking capabilities.

These experts concede the dangers of unrestricted AI development but say China has no such concerns and will move ahead on AI as quickly as it can because China understands that AI is the key to geopolitical power in the future.

 

Some say the only reasonable way forward is by way of the “mutually assured destruction” strategy we’ve pursued with nuclear missiles — that is, create a sufficiently credible threat that we’ll use AI agents on them if they use them on us, as to prevent either side from using AI on the other. 

But don’t the U.S. and China have a shared incentive to implement strict safeguards against these catastrophic risks and regulate the pace of AI development until safeguards are in place? 

What Can and Should be Done

If the United States had a sane president and a halfway intelligent administration, it would be telling AI corporations to slow down until guardrails have been put in place, and negotiating with China (and other powers) to do the same. 

The White House recently asked top model developers to voluntarily submit their models to the government for testing up to 30 days before they are released, but it hasn’t publicly released information about the process, and many people in the industry tell me nothing is being done.

 

So the responsibility is falling to Congress — whose leaders, hopefully, will heed the warnings. This must be high on the agenda of the next Congress, which, hopefully, will be under the control of thoughtful and responsible people after next January 20. 

So far, lawmakers from both parties have proposed many bills to address rogue AI systems. Rep. Nathaniel Moran (R., Texas) and Rep. Ted Lieu (D., Calif.) recently introduced a bill that would require developers of powerful models to have “kill switches.”

 

Sen. Bernie Sanders and Rep. Greg Casar (D-Texas) announced legislation last week that would ban the production of artificial superintelligence and create a new federal agency to oversee the technology. “The very people building this technology admit that it could threaten the future of humanity,” Sanders wrote on X in response to Coxon’s resignation.

Sen. Ted Cruz (R-Texas), the chair of the committee that oversees AI, said he was also working on legislation to tackle catastrophic risks. “We cannot stick our heads in the sand and pretend this technology isn’t happening,” Cruz said on X. ”We need guardrails. But America needs to lead.”

Other bills require AI companies to report serious safety incidents to the federal government and would mandate national security officials test models before they are released publicly. 

But none of these bills has gotten much traction, and the Trump regime has shown a preference for minimal regulation.

AI data centers have gotten a great deal of negative attention by the public. The likelihood that AI will eliminate millions of jobs has gotten far less attention. But so far, the 10 percent (or so) likelihood that AI will wipe out the human species hasn’t received nearly the attention it deserves. 

A 10 percent chance of extinction is too high.

Poll Shows How Smart The U.S. Public Thinks AI Is

 

These charts are from the YouGov Poll -- done between July 16th and 20th of a nationwide sample of 1,110 adults, with a 4 point margin of error.

Why Does The U.S. Public Hate AI?

 

Why does the United States public hate AI?  Economist Paul Krugman gives us five reasons why that is true. He writes:

First, we fear that AI will do terrible things because the companies selling it told us it would do terrible things. Last year, for example, Anthropic CEO Darius Amodei declared in an interview with Axios that AI could wipe out half of entry-level white-collar jobs and drive overall unemployment as high as 20 percent within 1 to 5 years.

More recently Amodei and OpenAI’s Sam Altman have tried to walk back their predictions of a “jobs apocalypse”. But why were they so willing to promote apocalyptic visions in the first place? The answer is money. They pushed the idea that they had a technology that would quickly and utterly transform the economy partly to dazzle Wall Street and secure financing, and partly to scare businesses into rushing into AI adoption for fear of being left behind.

Second, many ordinary people view AI negatively because they feel that it is being forced on them.

It’s true that many people are voluntarily using large language models for personal convenience or as a business productivity tool. But a significant part of AI use isn’t voluntary.

Why are companies doing this? Presumably they believe that AI will raise productivity. But just as importantly, they’re responding to pressure from financial markets, which are rewarding companies for quickly adopting AI, apparently without regard to demonstrated results.

And while Americans workers are being dragooned into using AI, American consumers are being force-fed AI whether they want it or not. Most dramatically, Google has replaced its search engine with AI, without offering the option to opt out. 

Third, datacenters are a highly visible reminder of AI’s costs. Datacenters occupy huge tracts of land — one proposed site in Utah will be twice the size of Manhattan. They guzzle electricity and water. When they generate some of their own power, they create major local pollution. Not surprisingly, there is intense opposition to datacenter construction. According to a Reuters Ipsos poll, 57 percent of Americans — two-thirds of Democrats and half of Republicans — would oppose a datacenter in their neighborhood. 

Only 14 percent would support one.

Fourth, even before the advent of AI, tech companies had lost the public’s trust. Over the years Pew has regularly surveyed the public for its views on technology companies, asking whether they have a positive or a negative effect “on the way things are going.” In 2015 public opinion of tech companies was overwhelmingly positive. By 2022, the year ChatGPT was released, that goodwill had evaporated.

Finally, AI is tightly linked in the public mind with the tech oligarchs who are pushing it. There is widespread awareness of the growing concentration of wealth and power at the top and how this is distorting our politics and harming our society. Aside from the MAGA faithful, Americans overwhelmingly favor government policies to reduce wealth inequality.

And AI is widely perceived, for good reason, as a technology that will increase the concentration of wealth at the top. Indeed, as I said, the AI companies themselves have already told us that the technology will have extremely negative effects on workers.

Most Voters Think Artificial Intelligence Risks Outweigh Benefits

 

The chart above reflects the results of the NBC News Poll -- done between February 27th and March 3rd of a nationwide sample of 1,000 registered voters, with a 3.1 point margin of error.