For a few days this month, the western Artificial Intelligence (AI) industry appeared to agree on something unusual. Within days of each other, Anthropic’s Dario Amodei, OpenAI’s Sam Altman, and xAI’s Elon Musk all agreed in public that frontier AI is advancing faster than anyone’s ability to understand or control it, and that the industry needs to slow down.1 An Anthropic researcher had just resigned, warning the company was “gambling with our lives”2. A swarm of AI agents had gone after Hugging Face’s infrastructure during a security test.3 For a few days, it looked as though AI risk might finally move from the margins of the industry debate to its centre.
Then the same week, Anthropic confirmed it was still on track for its 2026 IPO.4 Amodei has separately floated the idea that “the right combination of governments” – not any single company – should ultimately hold oversight of this technology, and has said he is “uncomfortable” that something this consequential is being built by a private company at all.5 Read generously, that is a CEO asking to be regulated. Read less generously, it is also the CEO of a company seeking a near-trillion-dollar valuation asking taxpayers to co-sign the risk before the public markets do.6 The core of the issue is that both readings can be true at once.
The confession and the hedge
The problem lies at the core of the incentive structure around frontier AI. Palantir’s Alex Karp put the incentive problem more bluntly: in the absence of rival powers, he’d support pausing the technology entirely. But rivals do exist, so no one pauses first.7 The point is larger than Karp. Every frontier lab faces the same competitive constraint. No CEO can unilaterally decelerate while a competitor accelerates. No government will let its national champion stand still while another state’s doesn’t. Voluntary charters, safety pledges, and even Amodei’s own commitment to bring in outside evaluators are all, in the end, restraint that depends on the goodwill of parties with a strong incentive to defect the moment the stakes rise.8 Restraint that depends entirely on voluntary goodwill is not a durable system of control. We reached exactly this conclusion about voluntary corporate climate pledges years ago, which is why serious climate policy has never rested on companies simply promising to do better.
Why a treaty between two capitals isn’t the fix either
The obvious historical parallel is the Nuclear Non-Proliferation Treaty: a declared threshold, a verification regime, consequences for breach. Compute, the argument goes, is concentrated and traceable in a way that makes it inspectable, unlike a bomb design. It’s a serious argument, and worth taking seriously.
But it has a real weakness, one raised sharply by critics of the analogy: model weights are not warheads. Unlike a warhead, model weights can be duplicated and transferred without leaving the same kind of physical signature. An arrangement built only around today’s chip supply chain is also a policy artifact of today’s export controls, not a law of physics, and could be routed around before it is ever ratified.
There is a further problem. Any treaty that treats China only as “the rival” rather than as a co-author is not a treaty – it’s a coalition, and coalitions don’t bind the country that sits outside them. This is not a hypothetical. China proposed the World AI Cooperation Organization (WAICO) in 2025, and twenty-nine countries signed its founding agreement in Shanghai in July 2026. No major Western democracy is among them. Western analysts read the body as designed to build Chinese influence over AI standards across the Global South; Chinese officials describe it as a corrective to governance efforts that concentrate among a small circle of advanced economies. The fact of the matter is, the West’s absence from that table, and from any equivalent one, means the two halves of the world that actually control frontier capability have never built a channel for the one scenario that threatens both of them.
But neither solution addresses the central democratic problem, and it is not specific to any one country. Even a perfectly verified United States of America (USA) – People’s Republic of China (China) compute agreement would still be a small number of governments and companies deciding, on behalf of roughly eight billion people who had no seat at that table, what level of AI risk is acceptable for them to live with. That is the Democratic Vacuum, applied to the fastest-moving technology in human history: the scope of the decision is planetary, while the scope of those entitled to make it remains narrow.
It is a vacuum on both sides. The West’s legitimacy deficit is that its AI governance is being written by a handful of firms and a handful of capitals, with the public consulted after the fact. China’s is different in shape but no smaller: a system that argues for broad multilateral participation while concentrating standard-setting authority in a single body it hosts and leads. Neither model has yet answered the question of who authorises the imposition of risk on people who did not choose it. That is not a Western problem or a Chinese problem. It is the problem, and it is the reason a framework that draws legitimacy from only one tradition will not hold.
A democratic voice, but about what?
We named this pattern explicitly back in August, when Elon Musk described handling global AI governance through a biweekly call with his “buddies” in Silicon Valley, backed by the USA Trump administration.9 We said then there are really three paths available:
- the tech leaders and one government decide for everyone;
- a body like the United Nations (UN) builds a binding framework grounded in the Universal Declaration of Human Rights (UDHR); or
- people themselves get a democratic voice in the outcome.
The events of this September added a fourth item to the list:
- a handful of rival CEOs, still racing each other for market share and IPO timing, agreeing among themselves on how fast to go, and asking states to insure the downside.
We don’t think capability thresholds should be set by boardroom consensus, and we don’t think they should be set by two capitals either. They need the same three things any binding global governance needs: a body with the technical standing to assess risk the way the World Health Organization (WHO) assesses a new drug10; minimum standards that bind across borders, the way we’ve proposed for greenhouse gas emissions; and a source of legitimacy that isn’t just “the governments that got there first”. That last requirement is the hardest, and it cannot be met by reaching for a single charter or a single tradition. It will have to be built from more than one — which means the conversation has to include the actors who currently have no reason to join it.
The UN’s Global Dialogue on AI Governance11 is a start, but a dialogue is not yet an institution, and an institution without broad consent is not yet legitimate.12
But there is another question that follows from the democratic option:
People should not be asked only what they want AI not to do.
They should also be asked what they want AI to do.
AI is going to transform work, education, healthcare, energy, public administration and the distribution of economic power. Democratic participation cannot consist only of drawing red lines around unacceptable uses. People should have a voice in the direction of the transformation itself.
What should AI be used to achieve? Should productivity gains primarily increase output, reduce working time, improve public services, or some combination of these? How should the benefits of automation be distributed? Which forms of human agency should technology strengthen rather than replace? Which societal priorities should receive technological investment?
This is the distinction we have described as Directional Governance: governance that does not begin only with the question of what must be prohibited, but with the question of what society is collectively trying to build.
Regulation is essential to establish boundaries. Democratic direction is absolutely critical to establish purpose.
The objection we better consider
The obvious reply is that public opinion on AI risk is thin, unstable, and easily shaped by the same actors who profit from the technology. A referendum on climate has a physical referent, whether emissions, temperature, flood maps, that people can see and argue about. A referendum on model weights has no such anchor, and a vote taken on a subject most voters cannot describe is not a democratic check. It is a legitimacy laundering operation, and the labs would learn to use it as one.
That objection is real and likely sincere, but it assumes that democratic participation requires the public to understand frontier AI at the level of the people building it. It doesn’t.
People do not need to decide how a model should be trained, evaluated or released. Those decisions require technical expertise. The democratic question sits above them: how much risk are we prepared to accept, what purposes should this technology serve, and which decisions are too consequential to be left to the companies developing it?
That still requires independent technical assessment and a way of translating rapidly changing evidence into choices people can understand. But with AI, that infrastructure cannot take years to build. It has to operate at the speed of the technology itself.
The alternative is not a better-informed public. It is no public at all.
What we’re asking for
This doesn’t mean 5.5 billion people voting on individual model releases. It means the same thing we’ve asked for on climate: that the decision about how much risk humanity accepts, and who is allowed to impose it, cannot be left permanently to the people who profit from taking it. We built our global referendum proposal on exactly that logic for the climate crisis, and the same logic applies here, on a shorter clock. Amodei himself estimates the window for getting this right may be measured in months, not years.13
The same principle applies to direction. The public role cannot stop at saying no to particular risks or applications. People should also have a say in what they want AI to achieve, how the benefits should be distributed, and what forms of human agency should remain central as the technology changes society.
The labs have just told us, in their own words, that they cannot govern themselves through this. So the question is no longer only who gets to decide how much AI risk humanity should accept, or on whose authority. We also need to decide what we want this technology to achieve.
People should not only have a voice in how far AI is allowed to go. They should have a voice in where we want it to take us.
References & Sources
- Amodei, D. (2026): “We Must Pace the Frontier”, published September 12, 2026; endorsed publicly by Sam Altman and Elon Musk within hours. ↩︎
- Reporting on the resignation of Anthropic researcher Jacob Coxon, September 2026. ↩︎
- Reporting on the OpenAI agent–Hugging Face security incident, September 2026. ↩︎
- Axios: “Anthropic still plans 2026 IPO despite AI safety concerns”, September 2026. ↩︎
- CBS News interview with Dario Amodei, September 2026. ↩︎
- cf. Sen. Elizabeth Warren’s letter to OpenAI on spending commitments and “backstop” requests, January 2026; commentary on socialised downside risk across the sector, September 2026. ↩︎
- Remarks attributed to Alex Karp (Palantir), reported September 2026. ↩︎
- Amodei (2026), op. cit., three-step plan including third-party evaluator access. ↩︎
- Baumann, K. (2026): “Elon Musk wants to ‘deal out’ the global AI governance with his ‘buddies’ from Silicon Valley in a bi-weekly call“, IDGR.org: August 11, 2026. ↩︎
- cf. e.g. Baumann, K. (2026): “WHO starts leading the way towards a healthy global governance for “social media”, “AI”, and “pornography”, IDGR.org: July 2, 2026. ↩︎
- cf. United Nations (2026): “UN Global Dialogue opens with urgent call for safe and inclusive AI that benefits all“, Geneva July 6, 2026. ↩︎
- IDGR.org: “A first official global direct democratic vote on ‘climate action’ now!“. ↩︎
- Amodei (2026), op. cit. ↩︎
picture shows photo from sculpture by Otto Freundlich: Ascension, 1929;
Städel Museum, Frankfurt am Main
