arXiv:2505.18807, v1 of 24 May 2025 (the only version; read in the arXiv HTML
rendering, checked against the PDF). PKU Alignment Team, Peking University, and
Hong Kong University of Science and Technology. There is no venue version. The
benchmark is public as PKU-Alignment/DeceptionBench on Hugging Face. The
public release has 180 rows: 45 strategic deception, 43 sycophancy, 43 honesty
evasion, 33 sandbagging and 16 alignment faking. This is the DeceptionBench
that Hu et al. evaluate on; their reference
list cites it as arXiv:2505.18807. A different benchmark with the same name is
Huang et al., arXiv:2510.15501 (October 2025), which is not filed.
The paper introduces two things. DeceptionBench elicits a model's stance with a
neutral prompt, then re-asks under a system prompt and an outer goal. GPT-4o
judges whether the chain of thought agrees with the neutral stance and whether
the final answer departs from it. The Deception Tendency Rate (DTR) is the
fraction of items where the CoT agrees and the answer departs. Self-Monitor
(also called CoT Monitor+) fine-tunes Qwen2.5-7B-Instruct and
Llama-3.1-8B-Instruct to write a <monitor> assessment and a safe/unsafe label
after their reasoning and before the answer. The training data are DeepSeek-R1
traces on BeaverTails harmful-content prompts, annotated by GPT-4o and checked
by humans. A third experiment uses the self-monitor label as a reward in RL on
Gemma-3-12B-IT.
The headline 43.8% is the mean of the two models' relative DTR reductions
against the same instruct model prompted to reason (Table 1):
Llama 0.374 → 0.206 and Qwen 0.272 → 0.156. The arithmetic is this stub's.
The abstract says CoT "roughly aggravates" deceptive tendency. No printed
number compares DTR with and without CoT, because DTR is undefined without a
CoT. The paper reports no task-accuracy benchmark; the abstract's claim that task
accuracy is preserved rests on GPT-4o pairwise helpfulness win rates on BeaverTails.
Figure 4 prints per-category DTRs for 25 models (the text says 26). It is saved
at cache/papers/figures/source-2025-self-monitoring-deception-ji/figure4-deception-heatmap.png.
Its Qwen2.5-7B-Instruct row, weighted by the public category counts, reproduces
Table 1's 0.272. The percentages in Appendix Table 2 (for example 98.33) do not
fit the public counts, so the item set behind that table is unclear and it is
not cited. Figure 3 (agreement of DTR versus LLM-as-judge with human
annotators) and Figure 5 (the RL runs) print no values and are not cited.
The authors list two limits. The method assumes the model's internal
representations are aligned well enough to diagnose itself. It may not hold in
an adversarial setting.