The US government is carefully watching AI companies. Every month, more and more DC insiders are waking up to the incredible amount of AI progress that may await us in the coming years, and how serious the implications of this will be.

On Wednesday, September 18, the US Senate Subcommittee on Privacy, Technology, and the Law held a hearing entitled “Oversight of AI: Insiders’ Perspectives.” Two of the invited witnesses were former insiders at OpenAI: 

• Helen Toner, a previous member of the OpenAI nonprofit board; 
• William Saunders, a previous member of the OpenAI alignment (and subsequently superalignment) team.

Their testimonies provided a rare insight into how former OpenAI affiliates view the rate of progress in AI, the incentives driving AI development, the strengths and shortcomings of OpenAI as a company, and the nature of the risks that lie ahead. 

Here are the five most important things we took away from the hearing: 

1. There is a gap between AI as everyday people see it today and as AI developers themselves see it. Insiders believe that progress will continue, rapidly, toward human-level capabilities.

Both Saunders and Toner emphasized that the singular goal of the handful of companies leading AI development is the creation of “artificial general intelligence,” or A.G.I. This would be a system that outperforms nearly all humans at nearly all economically valuable tasks — from data entry to scientific research to AI development itself. This is nothing like today’s AI, which helps write essays or answer college-level quizzes. 

A.G.I would transform the economy, and the world, unlike anything in history.

“In public and policy conversations, talk of AGI is often treated as either a science-fiction pipe dream or a marketing ploy. Among the scientists and engineers of these companies, it is an entirely serious goal—one that many involved see as achievable within 10 or 20 years, and some in as little as 1-3 years.” – Helen Toner

“When I was in high school, I spent years training for a prestigious international computer science competition. OpenAI’s new system leaps from failing to qualify to winning a gold medal, doing better than me in an area relevant to my own job. There are still significant gaps to close but I believe it is plausible that an AGI system could be built in as little as three years.” -William Saunders

2. AGI could pose catastrophic harms

Another point both speakers agreed on: human-level intelligence is a dual-use technology. We humans ourselves have used our intelligence for good and bad: progress in technology allowed both the eradication of smallpox via vaccination, and the flattening of entire cities via atomic weapons.

Human-level artificial intelligence will also be used for both good and bad. Should superintelligence — AI that far exceeds human capabilities in all domains — ever be developed, the scale of catastrophe we could face would be enormous. 

This is not purely hypothetical. We are already facing small, but real, catastrophic risks today. Chemical weapons, gain of function research, and cyberattacks already threaten social stability. Advanced AI will deepen this risk.

“AGI could also cause the risk of catastrophic harm via systems autonomously conducting cyberattacks, or assisting in the creation of novel biological weapons. OpenAI’s new AI system is the first system to show steps towards biological weapons risk, as it is capable of helping experts in planning to reproduce a known biological threat.” – William Saunders

“The problem is, the same people expecting to build advanced AI in the coming years believe that doing so could be incredibly dangerous. Experts agree that at a minimum, if we build AI systems that are smarter than humans in ways that matter, this technology will radically transform society. At a minimum, it will be an enormously powerful tool that could do huge harm in the wrong hands.” – Helen Toner

3. Companies like OpenAI are already dropping the ball on AI safety.

In order to prevent catastrophic outcomes from the development of AGI (and potentially even superintelligence), we need a herculean effort among AI developers to ensure safety and security. These few years are the dress rehearsal for AGI, and once it comes, we’ll have only one chance to get things right.

That’s why one of the most sobering (though perhaps not surprising) revelations in the congressional testimony was that AI companies are already dropping the ball. 

OpenAI in particular was called out for failing to meet their own bar for safety, as has happened in the past.

“It is jarring to see the mismatch between 1) internal expectations of how consequential these companies think the technology they are building will be, and 2) how seriously they are approaching questions about how safe their AI systems need to be and who gets to decide how they are developed and used.” – Helen Toner

“When I was at OpenAI, there were long periods of time where there were vulnerabilities that would have allowed me or hundreds of other engineers at the company to bypass access controls and steal the company’s most advanced AI systems including GPT-4.” – William Saunders

4. The entire AI industry is subject to fierce competitive pressure

When the world is developing a dangerous technology, race dynamics should terrify you. 

The most salient example of dangerous race dynamics was the development of nuclear weapons in the twentieth century — the global nuclear arsenal exploded from a total of 2 nuclear warheads in 1945 to over 63,000 in 1985, only forty years later. 

The primary cause of this was a competitive race between the United States and the Soviet Union. Both countries felt that their power would be threatened if they lost a technological and militaristic advantage over their adversaries, and nuclear proliferation began to spiral out of control.

International race dynamics aren’t the only way this can manifest. In the AI industry today, there are corporate race dynamics that are threatening our ability to proceed safely and responsibly. 

Ensuring that AI is used for good, and that catastrophic risks are adequately mitigated, will mean doing extensive research and safety testing. But this takes time — and when companies are constantly fearful about the prospect of losing out to their competitors, everyone begins to build and deploy frontier technology faster than they’d like. (Sidenote: these concerns are key to The Midas Project’s theory of change. Click here to learn more.)

“OpenAI will say that they are improving. I and other employees who resigned doubt they will be ready in time. This is true not just with OpenAI; the incentives to prioritize rapid development apply to the entire industry. This is why a policy response is needed.” – William Saunders

“Companies working towards advanced AI are subject to enormous pressure to move fast, beat their competitors to market, and raise money from investors. The current paradigm of AI development demands huge amounts of capital—billions of dollars, maybe tens or hundreds of billions—to build and run data centers containing hundreds of thousands of cutting-edge chips. There is also fierce competition over top talent, which companies can only attract if they are seen as the hottest place to do cutting-edge work.” – Helen Toner

5. There is no safe status quo — we are already exposed to significant risk

When considering approaches to mitigate the harms of artificial intelligence — safety research, policy responses, internal safeguards, etc. — there’s often a bias toward overweighting the costs of intervention, and underweighting the current costs of the status quo.

There are tradeoffs to everything. Inaction can carry just as many costs as action. What’s important is that we keep trying to make progress, even when it’s imperfect, and even when there are nontrivial tradeoffs associated with progress. 

If mitigating AI catastrophes means delaying the benefits of AI slightly, this is a deal worth making. The alternative is surely much worse.

“A ‘wait and see’ approach to policy is totally inadequate given that these systems are being built and deployed—and affecting hundreds of millions of people’s lives—even in the absence of scientific consensus about how they work or what will be built next.” – Helen Toner

“I resigned from OpenAI because I lost faith that by themselves they will make responsible decisions about AGI. If any organization builds technology that imposes significant risks on everyone, the public and the scientific community must be involved in deciding how to avoid or minimize those risks. That was true before AI. It needs to be true today with AI.” – William Saunders