Researchers on the forefront of artificial intelligence (AI) and leaders of many of the major platforms—from Geoffrey Hinton to Yoshua Bengio , Demis Hassabis, Sam Altman , Dario Amodei , and Elon Musk —have voiced concerns that AI could lead to the destruction of humanity itself.

Even the stated odds from some of these AI experts, with an end-days scenario as high as 25%, are still “wildly optimistic,” according to Nate Soares, president of the Machine Intelligence Research Institute (MIRI) and coauthor of the recent best-selling book If Anyone Builds It, Everyone Dies .

That’s because, as he argues in the book, the track we’re on with AI is headed for disaster—unless something radically changes. The book, cowritten with researcher Eliezer Yudkowsky, explores potential threats posed by “ superintelligence ,” or theoretical AI systems that are smarter than humans. “We’re sort of growing these AIs that act in ways nobody asked for, that have these drives and emergent behaviors nobody intended,” Soares said at last month’s World Changing Ideas Summit , cohosted by Fast Company and Johns Hopkins University in Washington, D.C. “If we get superhumanly intelligent AIs that are pursuing ends nobody wanted, I think the default outcome is that literally everybody on earth dies,” he added.

A reckoning for the world Likening the work of some AI leaders to building an airplane while flying with no landing gear, Soares said that not enough attention is being paid to the technology’s potentially negative outcomes.

The amount of global investment being poured into AI shows that people are betting it won’t be a “total dud,” he said, but there are two other “crazy” options: AI radically automates all human labor, so the economy is captured by a very small group, or it becomes super intelligent and kills everyone. “The world hasn’t really quite come to understand just how crazy this AI stuff is,” Soares said.

But there is some reason for optimism, Soares said, as a lot of people are worried about the future of AI, which makes for a “brittle” situation if more people—all of us included—voice their concerns. “Maybe if enough people are like, ‘Wait, we’re doing what now?

What the heck?’” Soares said. “Maybe that will shake the whole world into saying, ‘Holy crap, let’s change course.’”