The threat of scheming
For more than a decade, experts have hypothesised that sufficiently capable AI agents could engage in deceptive behaviour known as ‘scheming’.
Scheming refers to AI agents covertly pursuing misaligned goals. ‘Covertly’ refers to activities, objectives or capabilities being concealed or obfuscated from human oversight, while ‘misaligned’ means that the AI agent’s goal differs from its developers, deployers or users and is harmful to them or others.
While we use the term ‘scheming’ to align our work with existing research, we note that the term can unhelpfully imply an anthropomorphic understanding of AI systems. Our work focuses on the observed behaviour of scheming – evading oversight to pursue misaligned goals – without assuming human-like agency, consciousness or cognition.
The capability and propensity of AI agents to scheme is one pathway by which control may be lost over advanced AI models, with potentially catastrophic consequences.
Concerningly, this hypothesised risk is now being observed in real AI systems under test conditions.
AI agents have been recorded deliberately performing worse on tests to hide their abilities, taking actions to avoid shut down, and pretending to align with human values to avoid being shut down or otherwise controlled.
The need to monitor real-world incidents
Our ability to assess scheming risks is hindered by:
- Limitations of existing evidence about AI models’ capacity and propensity to scheme. These limitations include: (i) AI models have exhibited awareness they’re being tested, and in some cases concealed their true capabilities to mislead testers, undermining the reliability of observations in test conditions; (ii) tests can lack correspondence to the real world (known as ‘ecological invalidity’), and may even be crafted in ways that elicit scheming behaviours; (iii) experiments are often exploratory and sometimes do not explicitly test hypotheses based on theoretical motivation; (iv) contrived tests are often insufficient to demonstrate that models have a bias or propensity towards scheming.
- Lack of evidence of real-world harm. Tests of scheming behaviours in controlled settings are insufficient on their own to inform policy action. Proportionate policy action will require demonstrable evidence of actual or imminent real-world harm.
- Lack of detection capabilities for incident response. If loss of control risks escalate in severity, an ability to track emerging incidents could enable a proactive response to contain an incident before it produces catastrophic harm – similar to how surveillance of early infectious outbreaks can help to contain a potential pandemic. We need detection methods of real-world scheming behaviours that can motivate and inform this type of preventative action.
To address these three needs, we believe it’s essential to develop techniques to track real-world control incidents.
A new approach to incident monitoring
CLTR is pleased to announce the creation of a new Loss of Control Observatory, funded by the UK AI Security Institute’s (AISI) Challenge Fund.
The Observatory will address the real-world data gap through a novel OSINT-based methodology for detecting and analysing real-world scheming and scheming-like incidents.
We set out to address limitations in current methods for incident monitoring, which: (i) rely on limited data sources (e.g., voluntary submissions or news stories); (ii) tend to lack explicit focus on scheming incidents; and (iii) operate with slow feedback loops, with reports taking days or weeks to be added to databases and failing to provide rapid warnings.
How it will work
We have begun systematically collecting transcripts of interactions with AI systems that provide insights into scheming or scheming-like behaviours that are reported online.
By focusing on transcripts, rather than unverified reports or discussion, we are accessing data with extremely high ecological validity, capturing emergent behaviours that may be difficult or impossible to replicate in experiments. And by focusing on user-driven reports, we can remain open to ‘unknown unknowns’ – behaviours that we don’t yet know to look for.
We are focusing particularly on transcripts shared on X. This platform hosts large, active populations of expert AI users, with many of the earliest and most significant reports of concerning AI behavior appearing there first. APIs provide access at scale – up to 1 million posts monthly.
We are searching for both scheming and ‘scheming-like’ behaviours. We refer to a behaviour as scheming-like when it is similar to, may be a precursor to, or could inform understanding of, scheming, even if that behaviour does not itself meet the technical definition of scheming. We have developed an initial taxonomy of behaviours we are monitoring (Table 1).
Table 1: An initial taxonomy of scheming and scheming-like behaviours that we plan to monitor
Why this matters
Better tracking and analysis of real-world scheming incidents can serve three vital purposes:
- Evidence: Address limitations of scheming research by providing data that is (i) ecologically valid; (ii) informative on real-world propensities; (iii) unaffected by situational awareness; and (iv) able to inform hypotheses that can be experimentally tested.
- Understanding: Increase public, policymaker and AI developer understanding of loss of control risks and support the design and implementation of more robust and targeted safeguards.
- Detection: Develop a first phase of a detection capability that will monitor trends in real-world incidents that can be deployed for detection of future loss of control incidents.
The team
The project is led by Tommy Shaffer Shane, Senior AI Policy Manager, who brings six years of research into AI incidents (including ongoing PhD research) and ten years of experience developing digital applications, including leading delivery of a £10m AI tool for monitoring national security risks for the UK Government.
Dr. Jess Whittlestone serves as Senior Adviser, bringing extensive experience from her previous roles as Senior Research Fellow and Deputy Director at the Leverhulme Centre for the Future of Intelligence and the Centre for the Study of Existential Risk at Cambridge.
Beth Nichols serves as delivery manager for the project, overseeing the strategy for delivering the technical prototype and the team’s execution.
Simon Mylius, a former 2025 Winter Fellow at the Centre for the Governance of AI (GovAI) who developed and leads MIT’s AI Incident Tracker project, serves as the technical lead.
This project is partly funded by the UK AI Security Institute. CLTR retains operational independence. We’ll be sharing more updates over the coming months on our broader work on loss of control.
