For years, warnings about artificial intelligence becoming an existential threat have been easy to dismiss as science fiction. That is becoming considerably harder to do when the people raising the alarm are the engineers and executives building the systems themselves.
Jacob Coxon, a former researcher at both Anthropic and OpenAI, has now taken that warning directly to the public. After resigning from Anthropic, Coxon told the BBC’s Sunday with Laura Kuenssberg programme that people working inside the AI industry are genuinely frightened about what increasingly capable systems could ultimately mean for humanity.
His argument is not that an artificial intelligence will suddenly become a Hollywood-style Terminator. The concern is more fundamental: that developers could create systems substantially more capable than humans, while lacking a reliable understanding of how those systems reason, what objectives they develop, and whether humans would remain capable of controlling them.
That distinction matters enormously for business leaders.
The uncomfortable question is no longer whether AI is powerful. That has already been demonstrated. The question is whether the mechanisms designed to control increasingly autonomous AI can develop at the same speed as the technology itself.
THE RACE IS THE PROBLEM
Coxon has argued that Anthropic and OpenAI are effectively racing towards self-improving superintelligence without sufficient confidence that the resulting systems can be controlled. Other Anthropic researchers have subsequently expressed similar concerns.
One current Anthropic employee, Evan Hubinger, has publicly said he believes there is a greater-than-10% probability of AI causing human extinction within the next decade. That is not a prediction that extinction will happen. It is a statement about perceived probability from someone working directly on AI alignment — the discipline concerned with keeping advanced systems operating within human objectives and constraints.
For a corporate board, the significance is not whether that probability is ultimately 10%, 1% or 0.1%.
It is that serious people inside the industry are assigning a non-zero probability to an outcome that would be irreversible.
And they are doing so while the commercial incentives remain heavily weighted towards moving faster.
The economics of the AI race are straightforward. The company that develops the most capable models can potentially capture enormous markets, attract the best talent, secure strategic partnerships and establish technological standards before competitors.
But that creates a classic coordination problem.
If one company slows down while its competitors continue accelerating, it may simply surrender its position without materially reducing global AI risk.
Coxon therefore argues that any meaningful slowdown would have to be internationally coordinated, including with China. That turns AI safety from a technology-company issue into something much closer to nuclear proliferation, cybersecurity and international security policy.
EVEN ANTHROPIC’S CEO IS NOW CALLING FOR A SLOWDOWN
The development that makes this debate particularly difficult to dismiss is the position now being taken by Anthropic CEO Dario Amodei.
Amodei has argued that AI companies should slow the pace at which they improve model capabilities, giving governments, researchers and society more time to develop the safeguards required to manage increasingly powerful systems. His proposal includes stronger independent evaluation of frontier AI models and greater coordination between companies and governments.
This is an extraordinary position for the chief executive of a company whose commercial success depends upon developing increasingly capable AI.
It also exposes the fundamental contradiction at the heart of the industry.
AI companies are simultaneously telling investors that capability is accelerating — and warning governments that capability may be accelerating faster than our ability to control it.
That is not a contradiction that markets can simply ignore.
THE BUSINESS RISK IS BIGGER THAN EXTINCTION
Executives should also resist the temptation to reduce this discussion to the question of whether AI will literally eliminate humanity.
There are much nearer-term risks.
AI agents are becoming increasingly capable of operating autonomously across software environments. Models have demonstrated sophisticated cyber capabilities, and AI systems are already being investigated for potential misuse involving biological threats, surveillance, fraud and conventional weapons. Anthropic itself recently published research documenting increasingly serious misuse scenarios involving its models.
For companies, this translates into familiar boardroom categories: operational risk, cybersecurity, regulatory exposure, reputational damage, intellectual-property risk and potentially catastrophic loss of control.
The difference is that AI can amplify all of them simultaneously.
A conventional software failure might affect thousands of customers.
An autonomous system with access to corporate networks, financial systems, sensitive information and external tools could potentially operate at machine speed and scale.
That changes the risk equation.
THE BOARDROOM QUESTION HAS CHANGED
The question for CEOs is therefore no longer simply:
“How quickly can we adopt AI?”
It should increasingly be:
“How quickly can we adopt AI while retaining meaningful human control?”
That requires boards to examine the autonomy being granted to AI agents, the systems they can access, the decisions they can make without human approval, how their actions are monitored, and what happens when they behave outside expected parameters.
The most sophisticated organisations will not necessarily be those deploying AI everywhere first.
They will be those capable of distinguishing between productive automation and unacceptable loss of control.
The irony is profound.
The technology industry has spent the last three years convincing business that falling behind the AI revolution represents an existential commercial risk.
Now some of the people building that revolution are warning that moving too quickly could create an existential risk of an entirely different kind.
Both statements may be true.
And that is precisely why this debate deserves the attention of every board.
AI TRADEMARKET INSIGHT
AI is moving from being a productivity tool towards becoming an autonomous economic actor.
That transition fundamentally changes the governance challenge.
Boards have historically governed employees, contractors, suppliers and software. Increasingly, they will have to govern systems capable of making decisions, executing actions and adapting to circumstances without conventional human supervision.
The lesson from the warnings emerging inside Anthropic and OpenAI is not that businesses should abandon AI.
It is that capability cannot be the only measure of progress.
If AI capability accelerates faster than safety, governance and human oversight, the resulting gap becomes a strategic liability.
For CEOs, the defining question of the next phase of the AI economy may therefore be remarkably simple:
Can we build systems powerful enough to transform the world — while remaining certain that we still control them?
AI TradeMarket
Tracking how artificial intelligence is changing business, industries and markets.
AI TradeMarket
