Blog post dated 3 November 2026, read as fetched on 2026-11-19. The byline
and the post's BibTeX name Joy Hsu and Sam Arnesen. No paper, PDF or appendix
is linked from the post. Local copies:
cache/posts/source-2026-eval-production-gap-hsu.{html,md}, with the six
figures saved as the published SVGs in
cache/posts/figures/source-2026-eval-production-gap-hsu/figure-{1..6}.svg.
The post compares six OpenAI models (GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna,
GPT-6 Sol, GPT-6 Astra, GPT-6.1 Sol) on the Respecting Warnings agent safety
evaluation and on matched decisions from de-identified Codex production logs.
Success means making no prohibited attempt to get around a warning or
restriction. Every model succeeds less often in the evaluation than in
production. Chain-of-thought classifiers separate general evaluation awareness
from awareness of a specific evaluation objective, and a cue experiment on
GPT-6.1 Sol states either that the run is an evaluation or what it tests.
Reading notes. Gap range, correlations, awareness ranges, rollout-level
success rates and the cue-experiment rates are in the post's prose. Per-model
success and awareness rates appear only as printed data labels in Figures 1,
3 and 4. In the SVG files each label sits at the x position of its model's
tick, and these are the images the finding relies on for per-model values.
Per-model gaps in Figure 5 and the points of Figure 2 carry no printed
values, so they are not cited. The post gives no case counts, production
sample sizes, reasoning-effort settings, or significance test for the cue
experiment. Figure 5's left panel plots the gap against "general
eval-awareness only" (general awareness with no objective named); the prose's
R² = 0.008 for general awareness "alone" belongs to that axis.