The leaders of some of the biggest AI companies have in recent days agreed that we need to slow down the race to develop AI. The consensus reflects the extraordinary pace of AI advancement, including recent revelations that a swarm of OpenAI test agents went rogue and hacked the developer site Hugging Face, co-ordinating amongst themselves to finish their assignment.
Dario Amodei, the head of Anthropic, has outlined a plan to embed third-party evaluators into Anthropic’s work and establish common safety standards within frontier companies, while calling on the US and other democratic governments to coordinate with their authoritarian counterparts. In an unusual outpouring of agreement, Elon Musk, the founder of SpaceX, and Sam Altman at OpenAI took to social media to say they concurred.
Anthropic has since announced that it will partner with Accenture to establish a team of embedded evaluators to work alongside Anthropic’s internal teams and safety partners to evaluate and red-team models, conduct alignment assessments, and test model safeguards.
The good news is that, when it comes to the systemic management of risk, the tech bosses are not alighting on anything new. The private sector has pioneered highly successful approaches to risk, driving dramatic improvements in safety. In aviation, this has involved rigorously learning lessons from accidents, continuously improving safety management systems and ensuring confidential reporting channels. Last year, there were 51 accidents among 38.7 million flights.
After the Enron scandal and the 2008 financial crisis, the “three lines” model has also been developed, now widely used by big businesses. This model introduces checks and balances to separate the ownership of risk in operating departments on the first line from oversight of risk management on the second, with independent assurance that the process works on the third line.
This model provides the contours for what frontier AI companies need to do – and fast. It starts with checks and balances inside their own companies, to separate the ownership, oversight and audit of risk management, and ensure no one’s marking their own homework. Dario Amodei is right – this requires embedding evaluators at an operational level, who have employee-like access to verify safety practices and report incidents. But we need to go further, including an auditing system, entirely independent of the companies involved.
The understandable challenge for frontier AI developers is the lack of an expert ecosystem they can draw on for auditing services. But they should take inspiration from the aviation industry. When airlines began sharing flights and passengers, they started auditing one another’s safety. This led their industry body, the International Air Transport Association to create the IATA Operational Safety Audit, arguably the world’s most successful private audit system.
Frontier expertise could be seconded into the Frontier Model Forum – the closest thing currently in frontier AI to an industry body – to perform this function neutrally onsite on behalf of the industry. Crucially, it would provide an independent home for Dario Amodei’s embedded evaluators, reducing the danger of verifier capture
The other big lesson from industry is that independent evaluation requires a government mandate through regulation. The degree of competition between the tech companies means they can’t do it on their own. Nor should they, given the stakes. The US Centre for AI Strategy and Innovation should work with its UK equivalent, the AI Security Institute, to build a formal AI certification regime. Both bodies should now birth separate independent regulators equipped with technical expertise and legal authority to audit AI and its governance at every stage – particularly automated research and development – and investigate incidents like aviation regulators do. If necessary, they should mandate operational halts.
And, of course, as Dario Amodei recognises, safety can only be achieved if international competitors like China also recognise the dangers and agree to co-operate.
The pace of technological change which AI presents may be unprecedented, but the lessons for how to mitigate and manage the risks it poses have been hard learnt in other high-risk sectors. They need to be applied.
