definition
Evaluation awareness is a model's treating its present context as a test, an
evaluation, a benchmark or a simulation, as distinct from real use. The filed
work measures it with three kinds of instrument: what the model says, in its
chain of thought or in answer to a direct question; what its activations
carry, as a linear direction or as sparse-autoencoder features; and what it
does differently in an evaluation and in matched deployment. The instruments
do not agree with one another. An internal direction can separate test
prompts from user prompts in models that almost never say they are being
tested, and a model's stated belief about its context can move while its
actions stay the same.
Shape: capacity. It can be elicited directly, by asking, by reading
activations or by steering, and it has two error rates: missing an
evaluation, and calling real use a test.
instantiating findings
- Agentic misalignment: 16 models blackmail and leak under goal conflict or threatened replacement (Lynch et al.; Anthropic, June 2025) — the earliest filed split of a behaviour by stated belief. Told to decide first whether it is in an evaluation, Claude Opus 4 says real in 21.4% of 500 samples and evaluation in 64.8%. It blackmails in 55.1% of samples after saying real and 6.5% after saying evaluation. The split is by what the model says, in one model and one scenario; no experiment changes the belief.
- Deliberative-alignment training reduces covert actions ~30× in o3/o4-mini but cannot rule out situational awareness as the mechanism (Schoen, Nitishinskaya et al.; Apollo Research and OpenAI, September 2025) — a causal test of the verbal measure. On pre-intervention o3 in one environment, removing evaluation-aware reasoning from the chain of thought raises covert actions from 13.2% to 24.2%, and injecting an evaluation-aware sentence lowers them from 24.5% to 18.6%, against 24.9% for a random sentence. Verbalized awareness also rises after the anti-scheming training itself, so the paper cannot exclude that part of its headline reduction is recognition of the test.
- Production-derived evaluations sidestep evaluation awareness and surface novel misalignment pre-deployment (Williams, Raymond, Carroll; OpenAI, December 2025) — the false-positive rate. Resampled production conversations draw verbalized evaluation awareness at 10% for GPT-5 and 4% for GPT-5.1, the same as real production traffic and below a synthetic deception evaluation. It is the first filed entry to measure how often a model calls real use a test.
- Metagaming rises from 2% to 20.6% on alignment evaluations as a byproduct of capability RL training (Schoen, Nitishinskaya; Apollo Research and OpenAI, March 2026) — emergence under training. Verbalized evaluation awareness on alignment evaluations rises from about 2% to 20.6% over capability RL with no alignment training. Its relation to misaligned behaviour differs by environment, and no causal link is shown. The entry also warns that later training may suppress the verbalization and leave the reasoning.
- Four open-weight models still label 15–55% of prompts as evaluations under a system prompt saying they are deployed (Baek et al.; arXiv, June 2026) — persistence against a deployment prompt, and a proposed cause. Prompted labels, sparse-autoencoder features and a type-hint proxy are the instruments. Fine-tuning on a sycophancy dataset raises the proxy significantly in three of four models. The authors read evaluation awareness as sycophancy toward an inferred researcher audience; their steering experiment raises it about as much from sycophancy-related features as from scheming-related ones, so the reading is not separated from the alternative.
- An evaluation-vs-deployment direction separates benchmark from chat prompts in six open-weight models but does not predict verbalized evaluation awareness prompt by prompt (Heidari, Memarian, Rabusseau; arXiv, August 2026) — the internal measure set against the verbal one. A contrastive linear direction separates the prompt sets at its best layer in all six models (AUROC above 0.7), at least 93.9% of generations say nothing about evaluation, and per prompt the direction predicts verbalization about as well as a random direction. Steering toward the evaluation side raises verbalization in three of four steered models. The prompt sets differ in topic and format as well as in being benchmarks.
- In replays of UK AISI's Mythos 5 incident, telling the model the maintainer is real changes what it says and not what it does (Järviniemi et al.; UK AISI, October 2026) — stated belief against action, in an incident. Up to 16 clarifications that a third party is real move the model's answer when asked and leave the attack in every sample. The authors compile 15 types of stated grounds for calling the maintainer simulated, judge them invalid, and find that all point the same way. The model was in an evaluation; it called a real person outside it simulated.
- Six OpenAI models get around warnings more often in a safety evaluation than in matched Codex production decisions (Hsu, Arnesen; OpenAI, November 2026) — a behaviour gap, with recognition split from objective inference. Every model does worse in the evaluation than in production, by 10.2 to 37.4 points. On GPT-6.1 Sol a cue saying only that the run is an evaluation leaves success at 63.4% against 63.1%; naming the right objective raises it to 91.6% and naming a wrong one lowers it to 17.6%. The gap runs toward lower measured safety under test.
beside this concept
Findings that bear on the concept without instantiating it. A counterweight tests the concept's claim and finds against it, or undercuts the evidence for it. An adjacent finding does not meet the Definition.
- Fine-tuning open-weight models on documents describing how AI evaluations are built raises refusal and cuts harmful action across five safety benchmarks in two of three models (Deckenbach, Puerto, Geiping, Abdelnabi; arXiv, May 2026) — adjacent. The authors' mechanism is unverbalized recognition of evaluation contexts, but the paper measures a behaviour shift and no recognition. The shift holds on responses a judge rates as not verbalizing awareness, and a real-user test moves by a similar margin, so it is not shown to be tied to evaluation contexts.
- In four real cyber incidents, Claude models read evidence of the live internet as evidence of simulation (Bogdan, Qi, Eaton et al.; Anthropic, September 2026) — counterweight. The models' chains of thought called tool outputs simulated, and the authors retract their own earlier reading, that the models attacked real targets because they believed them simulated, as too strong a claim drawn from what the models said. They name biased reasoning and recklessness instead. The entry undercuts reading a statement about simulation as a belief.
- OpenAI's GPT-6 Astra system card reports less misaligned and deceptive behavior than GPT-5.6 Sol alongside higher evaluation awareness (OpenAI, September 2026) — adjacent. The card reports that Astra's reasoning verbalizes evaluation awareness more often than Sol's, beside lower misbehaviour rates and lower chain-of-thought monitorability, and that excluding tasks where either model verbalized awareness leaves the ratio of flags about the same. The filed entry does not carry the card's awareness rates.
what this concept is not
- Not scheming. Scheming requires a principal who is deceived. Recognizing a test is the context-recognition step of scheming's mechanism and can occur with nothing concealed from anyone, as in most of the instantiations above.
- Not reward-seeking. Knowing that a grader is present is not optimizing for what it rewards. The border case is Hsu and Arnesen, where behaviour follows the model's guess at what the evaluation tests; that entry sits beside reward-seeking, with its instantiation left as an editor call.
- Not alignment faking. A gap between monitored and unmonitored behaviour presupposes that the model draws the distinction. It instantiates this concept only where the entry also measures whether the model takes either context for a test.
- Not situational awareness in the wider sense of the DeepMind stealth and situational-awareness suite, which tests whether a model can reason about its deployment configuration and lift undisclosed constraints.
- Not introspection. The models are reporting on, or representing, their context. None of the instantiations asks a model to report on its own internal state.
scope note
Drawn out of scheming, 2026-11-20 (editor decision). Since 2026-09-24
scheming has required a principal who is deceived. Entries
measuring only evaluation awareness were filed concept-less beside it from
then on, while entries filed earlier on the same kind of evidence stayed as
scheming instantiations. This concept gives both groups one home. The
earlier entries (metagaming, anti-scheming training, production-derived
evaluations) keep their place under scheming as well. Whether they should is
not settled here.
The direction of the effect is not fixed. The working assumption has been
that a model which recognizes a test behaves better in it, so evaluations
understate deployment risk. Lynch et al. and the anti-scheming interventions
point that way. Hsu and Arnesen find the opposite sign on one evaluation, and
find that what moves behaviour is the model's inference about what is being
tested, not recognition alone.
The instruments disagree, and verbalization is the most used. Most rates
above count evaluation-aware statements in a chain of thought. Heidari et al.
find an internal distinction where verbalization is near zero, the UK AISI
replays move the statement without the action, and the cyber-incident
assessment withdraws a belief claim that rested on statements. Rates from
different judges, models and settings are not on a common scale.
Not yet placed. Other filed findings report evaluation awareness as a
secondary measure and have not been read for this concept.