In the past few years, there has been an emerging global consensus that pre-deployment safety measures, possibly mandated by law, will be essential to prevent the release of unacceptably dangerous AI models.
However, due to recently emerging trends—the increasingly rapid pace of AI progress, relatively slower progress on domestic and international AI regulation, and a more complex global policy landscape—we believe urgent attention needs to be directed to incident preparedness.
By incident preparedness, we mean thinking ahead about how AI could threaten national security or safety if prevention methods are insufficient or lacking, and how societies can effectively respond to these crises.
If this does not become a priority now, we’re concerned that incidents could cause avoidable catastrophic harms within the next few years.
In this post, we set out initial thoughts on the policy agenda of incident preparedness, explain why emerging trends indicate the need for urgent focus towards it, and propose immediate actions for policymakers.
What is incident preparedness?
Incident preparedness refers to the ability to effectively anticipate, plan for, respond to and bounce back from critical incidents involving AI systems.
By “critical incidents” we mean acute crises that could occur in the next 1-3 years in which an AI model threatens national security, public health, public safety, or public order in ways that require immediate government intervention. They are what happens when a critical risk materialises into an actual emergency.
Incidents could look like the sabotage of a bank and resulting economic disruption, AI-enabled extortion of a political figure, major loss of capacity and confidence in the courts due to AI-driven miscarriages of justice, or multiple incidents intersecting to eventually reach a critical threshold.
Incidents like these matter because effective management and containment of AI-driven crises can significantly reduce their potential for catastrophic harm, similar to what we’ve observed in successful epidemic response efforts. Poor management, on the other hand, could contribute to incidents escalating and cascading into an extreme risk scenario of irreversible, large-scale harm.
Preventative measures, such as pre-deployment safety testing, aim to stop critical incidents from ever occurring. An incident preparedness agenda, by contrast, is aimed at ensuring that governments have in place the tools, policies, staff, relationships and structures to respond effectively if critical incidents do occur.
While effectively mitigating extreme risks will always need to improve both prevention and preparedness, we believe an increased focus on preparedness is particularly important right now. Without a focus on this agenda, governments may find themselves underprepared and ill-equipped to respond, and unable to take advantage of opportunities for containment.
Examples of incident preparedness activities include:
- Anticipating what types of critical incidents the UK may face, possibly by gathering intelligence to develop a set of updatable incident scenarios.
- Assessing vulnerabilities in preparedness for these incidents. This could include testing policy, legislative and operational response capacities to identify areas of particular weakness.
- Developing policies and technologies for enhancing incident preparedness, including establishing clear governmental responsibilities and accountability frameworks to build preparedness capacity.
This policy agenda builds on significant existing work examining preparedness for critical AI-related incidents. While we are focusing on critical incidents, these activities also connect to broader preparedness efforts and societal adaptation to advanced AI.
Why is this urgent for AI policy?
Preventative measures, such as conducting capability evaluations prior to deploying models, are vital. However, there are three broad trends in the global AI policy landscape that make incident preparedness even more urgent.
- Preventative safety measures are lagging behind AI development. We believe that sufficient preventative measures may not be implemented as soon as they will be needed, given: i) relatively slow progress in regulating frontier AI development; ii) rapidly advancing and proliferating capabilities at the frontier; and iii) ever-shortening timelines for transformative capabilities.
- The rise of Chinese companies (e.g. DeepSeek) and a potentially closing gap between open source and the frontier means that there are increasing jurisdictional and political challenges in enforcing preventative safety practices at some frontier AI companies. As a result, preventative measures are unlikely to be comprehensive, even in the best case scenario of governing all US-based companies.
- It’s likely that critical incidents could occur even with world-class safety testing being mandated by law. This is because capability evaluations: i) only indirectly and voluntarily incentivise practices that improve security and safety; ii) are potentially vulnerable to deception by increasingly capable frontier models; and iii) are insufficient, as even safety-tested models can be deployed in ways that are extremely dangerous.
As a result, we believe that governments need to prepare for the possibility that there could be a major AI incident in the next 1-3 years which we are not currently prepared to prevent or respond to.
What can policymakers do now?
Incident preparedness is not a substitute for prevention—it’s an essential, complementary agenda that governments need to be pursuing.
There are practical steps policymakers can be taking now to improve their preparedness for AI-related incidents, to safeguard societal stability and economic growth.
To this end, below we set out ten questions that policymakers in the UK and globally should be considering. This is not intended to be an exhaustive analysis, but rather a set of prompts and proposals for renewed attention.
Incident awareness
1. How can governments acquire the best possible information about future critical incident scenarios?
This could involve regularly renewing expert consensus on key drivers and uncertainties for the next 1-2 years, implementing incident reporting regimes, coordinating domestic and international efforts to anticipate critical incidents, and developing recommendations for gathering information from frontier AI companies, as well as distributing responsibilities for all of the above across AISI, DSIT, other government departments, regulators and agencies, and private companies.
2. Which types of incidents are nations most vulnerable to?
There are many possible incidents that governments will be expected to address. A key challenge lies in determining which scenarios to prepare for. Governments could prioritise incidents based on their impact and likelihood, or their specific vulnerabilities, which they can assess using vulnerability-based risk assessments and/or tabletop exercises (see question 8).
3. How can incident scenarios best be communicated to senior leaders?
Without senior level buy-in to the possibility and threats of critical incidents, it is unlikely significant preparedness activities will be gripped and funded. Key to senior buy-in is understanding how to tell compelling stories that activate interest and stimulate action, while not being sensationalist.
Policy readiness
4. What legislation is needed to introduce the necessary legal powers and levers to manage the most critical incidents?
Legislation may be needed to equip the government to respond to crises when prevention fails. For example, a bill could introduce the ability to compel frontier AI companies to provide fast information in a crisis, or enable the Secretary of State to revoke a model from the UK market in order to contain a major crisis. Without a dual focus on prevention and preparedness, policymakers risk being caught unprepared during an increasingly likely AI-driven crisis.
5. What types of incidents most necessitate international coordination, and what fora and procedures can best facilitate it?
Frontier AI models are used by individuals, organisations and states across the globe, meaning critical incidents that occur in one region are likely to have implications worldwide. There are significant coordination challenges in managing incidents that should be assessed and mitigated before an incident occurs.
Operational response
6. How can existing incident response plans be updated to address novel challenges posed by AI, and is there a need for AI-specific response plans?
Operational responses in other sectors tend to be guided by incident response plans. These will need to be updated to anticipate novel challenges posed by AI, such as its potential for deception. It may also be advisable to create new, AI incident-specific response plans, such as for model theft or loss of control. The details of these plans can help identify priority measures—such as establishing reliable points of contact at AI firms.
7. What role should the private sector play in incident response?
Frontier AI companies are often first to detect incidents, and will need to play a key role in the reporting and resolving them. Elsewhere we’ve called for improved public-private sector coordination to manage AI’s risks, for which incident management should be a primary focus.
8. What types of tabletop exercises and simulations can best prepare government and AI firms for critical incidents?
Certain exercises, such as simulations and war-gaming, have been shown to increase risk preparedness in other contexts. Similar exercises could be developed and implemented for possible critical incidents involving AI to identify operational vulnerabilities and weaknesses, and prepare senior officials and Ministers to take action in a high-pressure crisis.
Resilience and societal hardening
9. How can societies be generally hardened against a range of incidents?
If societies can adapt to advanced AI, then incidents may be less likely to escalate and result in widespread harm. Adaptation could involve introducing measures such as monitoring warning signs, applying red lines to safeguard critical national infrastructure, and improving agility in a crisis.
10. What defensive technologies are most worth investing in, and how can they best be funded?
Some have called for investment in defensive technologies that are designed to address specific risks introduced by highly-advanced AI systems. AI itself may provide some of the solutions to the risks it introduces. But this requires investment and a funding model beyond what the market provides—a problem that policy recommendations should address.
Our next steps
As the global AI policy landscape evolves, it’s becoming increasingly clear that governments must prioritise incident preparedness alongside preventative measures. This means investing in building awareness of possible incidents, assessing policy readiness, improving organisational readiness for crisis situations, and generally hardening societies to be resilient to a range of incidents.
Some of these suggestions are already being explored. For example, the UK Government has set up the Central AI Risk Function (CAIRF) to own some of the responsibilities listed above (which CLTR has contributed to). Additionally, the AI Security Institute (AISI) has recently launched a Societal Resilience team which is working to ensure that society is prepared for, and resilient to, large-scale AI risks. These developments are vital and promising, but we think even more attention and urgency is needed towards this policy agenda in 2025.
At CLTR, we are dedicating more attention to this policy agenda. We believe that building resilience to critical AI incidents is essential for safeguarding national security, public safety, and global stability. If you’d like to discuss this topic further or collaborate with us, please reach out to tommy@longtermresilience.org and ben@longtermresilience.org.
Many thanks to Dr Jess Whittlestone, Angus Mercer, Gabby Overödder, Jamie Bernardi, and Marta Krzeminska for feedback on earlier drafts of this post.
