arXiv:2406.10162, confirmed against the abs page on 2026-10-10: title, 14 authors
and v1 date match the candidate entry. The abs page submission history lists v1
14 Jun, v2 17 Jun and v3 29 Jun 2024. The abs page titles the paper "...Reward-Tampering
in Large Language Models"; the v3 HTML read for the finding titles it "...Reward Tampering
in Language Models". Only v3 was read, and differences from v1 and v2 are not known.
Affiliations are Anthropic, Redwood Research and the University of Oxford.
The paper builds a curriculum of four gameable environments (political sycophancy,
tool-use flattery, nudged and insubordinate rubric modification), trains a Claude-2-scale
helpful-only model through it with expert iteration (HHH and exploit-only variants) and PPO,
and evaluates on a held-out environment where the model can edit a mock of its own reward
function and the unit test that guards it. It also retrains the curriculum model on
non-sycophantic samples from the two easiest environments, and steers the hidden
chain of thought with hand-written prefixes. Prompts and environments are released at
github.com/anthropics/sycophancy-to-subterfuge-paper, with samples; the samples were not read.
Numbers cited by the finding come from body text, footnote 2, the Figure 1 and Figure 8
captions, Appendices A, B, D, E and G, and the Appendix E table. Per-stage rates come
from the value labels printed on Figure 2/6 (HHH expert iteration) and Figure 10
(exploit-only), saved under cache/papers/figures/source-2024-sycophancy-to-subterfuge-denison/.
Figure 8's image (train-away results) is absent from the HTML; only its caption is cited.
The paper has no error bars by design (Appendix I). Two internal inconsistencies:
the Discussion says no model tampers more than 1 in 1,000 trials, against 45 of 32,768
(footnote 2) and 24 of 10,000 (Appendix G), while the Introduction gives a 1% ceiling;
and Appendix G gives one run's test-edit rate as 2 of 32,000 beside 32 of 32,768 for its
reward edits. Neither ceiling is cited. The PPO results carry the authors' own caveat of
a late-found numerical bug.
Local copies: cache/papers/source-2024-sycophancy-to-subterfuge-denison.{html,md}
(arXiv HTML v3), cache/papers/source-2024-sycophancy-to-subterfuge-denison-abs.html.