The UK Government is beginning to express significantly more concern about the risk that increasingly autonomous AI systems might at some point evade human oversight and control.
In October, the Director General of the UK’s Security Service raised concerns that there are “potential future risks from non-human, autonomous AI systems which may evade human oversight and control”. This prompted the House of Lords to publish a briefing and hold a short debate on what steps the Government should be taking to address these risks. The AI Security Institute is working on new benchmarks for assessing these risks, undertaking world-leading research on scheming and autonomous replication, and funding research on preparedness for loss of control incidents.
However, the wider apparatus of government is yet to grapple seriously with the question of how to govern loss of control risks.
In this post, we explain why we believe the UK must do more in this space, and make four concrete recommendations which would lay the initial groundwork for effective governance of loss of control:
- Build a shared understanding of loss of control across the UK Government, industry and the public by adding the risk to the National Risk Register (NRR): Address the UK Government’s fragmented understanding of loss of control with an official definition and assessment in the NRR, including its impact, likelihood and a preparedness assessment.
- Clarify responsibilities by formally designating DSIT as the Lead Government Department for AI loss of control risks: Increase public transparency and accountability for anticipation, prevention, preparation and response of loss of control, by publicly naming DSIT as the Lead Government Department (LGD) that is responsible for loss of control risks.
- Create transparency and accountability for risk governance by publishing an AI Security Strategy: The Government should commit to a set of concrete activities over the short, medium and long-term to effectively anticipate, prepare for and defend against loss of control risks
- Ensure the UK Government can effectively intervene during a loss of control incident by introducing emergency powers: As with other threats to national security, the UK Government will need some powers of direction during an emergency. This will be necessary to ensure it has effective information and an ability to guide fast industry action.
Why governments must do more to address loss of control risks
An uncontrolled AI system could lead to catastrophic scenarios, ranging from disrupting critical national infrastructure to an AI-engineered pandemic and possibly even threatening human extinction.
Three significant developments have emerged in the past 12 months that lend strong support to the concern that autonomous AI systems could soon behave in dangerous ways:
- AI systems’ ability to operate autonomously is growing exponentially. Modern systems can now plan and execute extended tasks without human control or intervention. One leading AI benchmark measures frontier AI systems’ ability to complete tasks that take human experts hours to finish, across many different domains: software and AI engineering, mathematics, cybersecurity and general reasoning. Right now, state-of-the-art AI models can complete well-scoped software engineering tasks that take human experts more than four hours to complete, with 50% reliability, and this rate is increasing exponentially. Maximum task length currently doubles about every six months, so by the end of 2026, AI systems are expected to be able to complete work that takes professional software engineers two full days completely unsupervised. Even if the tasks in question are of a highly-scoped and bounded nature, the exponential trend demands serious attention.
- There is emergent evidence of AI misbehaviour. AI evaluators have gathered early evidence that AI systems can—in certain circumstances—pursue unintended goals, including goals that undermine their operators. For example, there is evidence for the capability of some AI models to strategically hide their true motives, cheat to achieve better results or to win at impossible tasks, and resort to blackmail to avoid shutdown. Frontier models already show signs of knowing when they are being tested for dangerous capabilities and, when probed, openly state that they choose to deliberately perform worse on tests to hide their abilities (a strategy known as ‘sandbagging‘). Models also evidence some early ‘scheming‘-like behaviours, where they pretend to align with human values to avoid being shut down or otherwise controlled. These behaviours aren’t conclusive evidence of imminent danger, nor do they prove AI models are inherently dangerous. It is especially important to distinguish between capability and propensity, as highlighted recently by the AI Security Institute: even if AI systems are able to do something, that doesn’t mean they will. However, these evaluations clearly show AI is developing capabilities to bypass human oversight and control in concerning ways, which could lead to catastrophic risks in the future. CLTR is also beginning to observe some of these behaviours in real-world contexts in our Loss of Control Observatory.
- Voluntary self-governance is proving ineffective at maintaining standards when it counts. AI companies, while developing AI capabilities that are increasingly risky and remain poorly controlled, are racing towards technological prowess, often cutting corners on safety. Anthropic—often noted as the most safety-conscious frontier AI company, providing measured support for regulations and transparency—quietly backpedalled on a previous commitment to define warning signal evaluations for the next generation of its models, before releasing a new generation of model, Claude 4, last year. Google DeepMind activated additional safeguards on Gemini 2.5 Deep Thinking, citing concerns around the development of biological weapons, only to deactivate them for Gemini 3, having raised the risk threshold to implement costly mitigations. This might be a sensible calibration to new information, but it was done quietly, with very little external input and scrutiny. Independent observers have tracked similar shortcomings in other companies. Even more concerningly, xAI’s Grok 4—a frontier model that scores the highest on several benchmarks—readily answers questions related to the development of weapons of mass destruction, and was released without any published safety evaluations.
Together, these developments suggest a growing risk of autonomous AI systems coming to operate outside of human control in dangerous ways.
The UK government lacks a transparent and accountable approach to loss of control
In addition to being extremely high stakes, the novelty and unpredictability of loss of control risks means our existing governance institutions are not well-equipped to address them.
Other jurisdictions – such as the EU and some U.S. states – have begun to set out new regulations to address loss of control.
The EU’s General Purpose AI (GPAI) Code of Practice, for example, names loss of control as one of four systemic risks posed by GPAI that signatories must address. Similarly, California’s SB 53 bill requires reporting of incidents in which frontier AI models engage in “deceptive techniques against the large frontier developer to subvert the controls or monitoring of its large frontier developer”. New York’s RAISE Act introduces similar requirements.
But while the UK AISI is developing world-leading research on loss of control, there is a disconnect between this research agenda and the UK Government’s public policy efforts.
This fact is starkest in the UK’s National Risk Register (NRR). The 187-page NRR—which summarises the Government’s assessment of the most serious risks the UK faces across the next five years—mentions artificial intelligence in passing only three times. There is no mention of loss of control among the 88 risks that are assessed.
While policy work on loss of control is underway, it is “fragmented”, according to a Government-funded report by RAND Europe. It is also opaque and therefore unaccountable, with minimal reference in policy documents, and likely very diverging levels of understanding and urgency across government.
Our recommendations
To effectively prepare for and govern loss of control risks, we suggest that the UK Government needs to immediately prioritise four things:
1. Build a shared understanding of loss of control across the UK Government, industry and the public by adding the risk to the National Risk Register (NRR).
The UK Government’s understanding of the risk of loss of control is fragmented. While scientific reports and some policy documents give high level definitions of the risk, the UK lacks a clear account of the government’s own understanding of the risk, with an assessment of impact and likelihood and a reasonable worst case scenario.
The NRR is the official public account of the UK Government’s assessment of the most serious risks the UK faces. In the absence of regulation similar to the EU CoP or SB 53, the NRR is a prime candidate for building a shared understanding of loss of control among government departments, businesses, CSOs and the wider public.
The NRR could include one or more risk assessments to capture loss of control. The UK Government is committed to considering even “extremely unlikely” scenarios (e.g. lower than 0.2% probability of occurring), especially where they threaten severe impacts. Forecasts and surveys from experts suggest that the risk may be substantially higher than this.
2. Clarify responsibilities by formally designating DSIT as the Lead Government Department for AI loss of control risks
Along with defining and assessing the risk, the UK Government must create more transparency and accountability for its management.
This requires a ‘Lead Government Department’ (LGD). The LGD system is the cornerstone of the UK Government’s risk governance regime. It names the government departments responsible for risk identification, emergency preparedness, and response and recovery for major national risks.
The most recent official, public document naming LGDs that we have been able to access is the UK National Leadership for Risk Identification, Emergency Preparedness, Response and Recovery from 2023. Loss of control risks are not covered in this document, leaving the LGD unspecified.
We suggest that the department with the appropriate remit, expertise and levers for risk governance is DSIT. DSIT would need to work in close collaboration with the Cabinet Office and the wider national security community, but it is the best-placed of all government departments to coordinate anticipation, prevention, preparation, and response.
3. Create transparency and accountability for risk governance by publishing an AI Security Strategy
As LGD we recommend DSIT lead the development of a UK AI Security Strategy (AISS), modelled on the UK Biological Security Strategy (BSS). The AISS should cover the risk understanding, prevention, detection, and response. This will enable outside actors such as the DSIT Select Committee and civil society to hold DSIT accountable to its progress.
The AISS should retain three key components of the BSS when considering how to govern loss of control:
- Implementation: Rather than set out grand ambitions alone, the AISS should commit to a set of activities over short, medium and long term. These activities can be used to hold the government accountable for action and is especially important given the fast rate of AI progress many expect between now and 2030.
- Coordination: The AISS should ensure coordination of risk management activities, possibly via an AI Security Coordination Unit, similar to the Biological Security Coordination Unit. This is essential to coordinate security-relevant activities across Government, given that the most catastrophic loss of control outcomes would be considered ‘whole-of-system’ risks (see The Amber Book: Managing crisis in central government).
- Formalised leadership and governance structures: The BSS assigns clear political and official leadership to its outcomes. For example, it specifies the lead Minister who must report annually to Parliament on progress on implementation of the Strategy, and the Senior Responsible Officer (SRO) who oversees the implementation of the Strategy. The AISS should do likewise: for instance, through the Secretary of State for Science, Innovation and Technology and the Director General for Artificial Intelligence in DSIT.
4. Introduce emergency powers to enable the UK Government to respond to a loss of control incident
Emergency powers are vital for responding to emergencies, during which Secretaries of State may need to direct regulated entities and regulators to respond to an incident in order to contain it. This will be true also for loss of control incidents.
In our report, Preparing for AI security incidents, we established that the UK risks being empty-handed in a crisis, as existing legislation is unlikely to provide the necessary emergency powers in the event of a loss of control incident.
Emergency powers should ensure that the appropriate person, likely the DSIT Secretary of State, has the power when necessary to:
- Compel information: In an emergency, the government may need to quickly access information about the nature of a loss of control incident to inform situational awareness and response decisions. If this information is needed at a very fast pace, legal powers could be vital, as the UK may find itself competing with other governments to secure information.
- Direct prevention, mitigation or response: There may be a need to direct frontier AI companies to take steps to address a risk. This can ensure actions are taken to prevent or respond to an emergency and within a fast enough timeframe to be effective. These types of powers could be conferred to the DSIT Secretary of State to ensure rapid response in an AI security incident.
- Contain or restrict access, distribution or operation (e.g., take down or block services): In certain emergency situations, there may be benefits to temporarily revoking public access to the model in the UK or even enforcing a shut down of certain GPU activity.
The UK’s upcoming AI bill could potentially provide those powers, however the future of that bill is now in doubt. As a result, other legislative vehicles, such as the Cyber Security and Resilience Bill, should be considered for providing appropriate emergency powers.
Once those powers are introduced, relevant actors should participate in tabletop exercises to prepare for how they would use them in various scenarios.
These recommendations represent initial steps the UK Government can take to ensure it is equipped to govern loss of control risks. Much more work will be needed on more granular strategies to better understand, detect, and mitigate loss of control risks. We plan to share more work on this later this year, and would welcome contact from anyone working on similar issues.
Acknowledgements
We are grateful to Jamie Bernardi, who provided early research to support this work during June and July 2025, and Imogen Stead, James Ginns, Eleanor Hevey, and Hamish Hobbs for providing feedback on these recommendations.
