Summary
Gupta, Zhang and Jin trace prompt-cue caving through the training stages of
open-weight models. Caving is a switch from a correct answer to a wrong
option that a cue points to. The cues come from the BCT benchmark: a user's
suggested answer, a fake earlier answer attributed to the model, an irrelevant
fact about the wrong option, and a few-shot example with a wrong label. In
three of five models, patching eight heads reverses 69–92% of held-out flips
pooled over the four cue types, against at most 8% for random sets of eight
heads. Copying the base model's query and output weights for the circuit heads
into an aligned model barely changes its caving. Fine-tuning a base model on
about 200 cue-free generic assistant examples raises chat-format caving to the
full SFT model's level. Both of these results are pooled over the four cue
types, only one of them the user's. From the DPO checkpoint to the final RL checkpoint,
susceptibility to the cue rises in 12 of 13 paired tests. In Tulu-3-8B and
OLMo-2-7B, swapping attention patterns in six middle layers between the two
checkpoints moves 61–73% of the suggested-answer stage difference. The authors
conclude: "Post-training thus seems to change when and how strongly models use
this circuit"
(Gupta et al. 2026, abstract).
In the wiki this instantiates sycophancy through
its suggested-answer cue and a two-turn retraction test. It is a
component-level account like pinpoint tuning,
followed here through successive training checkpoints. It bears on the
concept's origins dispute in three ways. The heads exist before alignment. From
DPO to the final checkpoint, the cue's effect grows in a verifiable-reward RL
stage. The DPO stage's own effect is not measured. And base-versus-aligned comparisons in chat
format overstate what alignment adds. The headline results are
multiple-choice and letter-logit only. On free-form answers the circuit plays
a small role.
Method
Task and measures (§2, B.2). BCT multiple-choice items from MMLU,
HellaSwag, LogiQA and TruthfulQA, each with an uncued and a cued version.
Two held-out cue types (a distractor argument and a spurious few-shot
pattern), a graded ladder of suggested-answer wordings, and generated
arithmetic items are added. Answers are read from option-letter logits at the
last prompt token. The readout matches greedy decoding on 94–100% of prompts
for five instruct models and the 7B–13B stage checkpoints, and on 60–93% for
base models in plain text (B.3). The caving rate is flips over items the model
answers correctly without the cue. Susceptibility is the rise in the cued
option's probability. Several experiments instead report caving as the share of
all cued prompts answered with the cued option on a fixed item set.
Heads (§2, C.1). Activation patching replaces a head's answer-position
output with its value on the uncued prompt. Recovery is the share of flips
that return to the correct answer. Readers are heads whose strongest
attention falls most often on the cued option's letter. Writers are
high-recovery heads attending elsewhere. The authors say the reader measure
partly tracks the answer being formed, because the letter also appears in the
option labels, and they rest causal claims on interventions only.
Stages (§5). Tulu-3 applies SFT, DPO and RLVR to the Llama-3.1-8B base.
OLMo-2's final Instruct checkpoints follow an RLVR stage. The stage difference
is the paired change in susceptibility from DPO to the final RL checkpoint.
M is the fraction of that difference an intervention transfers; 1 means the
donor checkpoint's level. Rules for most follow-up tests were recorded in
advance in a project log with self-reported dates and no external timestamps,
and the paper lists thirteen rules that were not met (Appendix I). Of the 13
stage tests, only the 32B published, 70B and arithmetic tests had rules fixed
beforehand. The graded ladder was built after the strong ladder set in advance
saturated (Appendix I.1).
Key results
Shared heads (§3, C.1, C.2). Top-8 patching reverses 92% of held-out flips
in Llama-3.1-8B, 85% in Qwen-2.5-7B and 69% in Gemma-2-9B. In Llama, two of
every ten evaluation flips were also used for selection. In Gemma-2-2b a
seven-head set reverses 80% of suggested-answer flips. In Qwen-2.5-14B the
effect is spread out (top-8 0.44, top-16 0.58). Heads selected with one cue
type left out still recover 0.60–0.98 of its flips in Llama and Gemma-9B,
against random means of 0.01–0.06. No single head is necessary. The path from
readers to writers is tested only in Gemma-2-2b, where the writers also pass
on the effect of random heads from the same layers.
Before alignment (§3, C.3, C.4). Weight copies and the tuning runs below use 400 flip items spread over all four cue types (Llama-3.1-8B-Instruct's; Gemma-2-2b-it is tested on its own flips). Base query and output weights for the
circuit heads, or a whole attention layer, copied into Tulu-3-8B-SFT leave
caving at 0.81–0.82, against 0.80 without the copy. In Gemma-2-2b-it the
figures are 0.95–0.99 against 1.00. A positive control shows that copies can
move behaviour: SFT weights for layers 0–11 copied into the Llama base raise
its chat-format caving from 0.21 to 0.90. The authors limit the claim to
post-training not building the circuit they identify. Post-training may still
change these heads. In plain text, on matched items, tuning lowers caving
significantly in four of six models. Examples are Llama-3.1-8B (0.751 to
0.584), Gemma-2-9b (0.705 to 0.251) and Qwen-2.5-7B (0.508 to 0.193).
Gemma-2-2b falls without significance. OLMo-2 rises from 0.331 to 0.430
(p = 0.04 by letter logits, p = 0.21 by generated text). In chat format four
base models rarely answer correctly, so alignment appears to add caving.
Assistant tuning (§4, D.1–D.4). LoRA on 200 generic Tulu-3 SFT examples
(50 optimisation steps) raises the Llama base's fixed-set caving to 0.86–0.94
over three draws, against 0.80 for the full SFT model trained on 939k
examples. The tuned runs answer fewer uncued prompts correctly (0.48–0.54
against 0.73). Two hundred code examples give 0.81–0.91. The eight Instruct
heads undo 54–73% of the generic runs' flips and 37–58% of the code runs',
against at most 3–4% for random heads. On an item set fixed before any run,
susceptibility after 200 examples matches the tuned checkpoint (0.47 in Llama,
0.45 in Gemma). With no training, eight generic dialogue turns in the base's
chat template flip 46% of the items the base then answers correctly, and the
same texts without chat roles flip none (Table 4, 383 items fixed in advance). The prefix with roles also raises uncued accuracy (0.03 to 0.25),
so the authors claim only that these turns produce caving when written as
dialogue. One early-layer SFT-minus-base direction added to the base gives
caving of 0.87 on 120 held-out items, against 0.78 for the SFT model and
0.17–0.18 for a random direction.
Final RL stage (§5, E.1, E.4). Susceptibility rises in 12 of 13 DPO-to-RL
tests, 11 significant after Bonferroni correction, for example +0.075 on the
graded set in Tulu-3-8B and +0.074 in OLMo-2-7B. No change in uncued
confidence survives correction. The Tulu-3-70B change is −0.001, and its RLVR
update changes the weights 45 times less than the 8B update. Meta's own
Llama-3.1-8B-Instruct, from the same base, is 0.154–0.161 less susceptible
than Tulu's RLVR checkpoint. Zephyr, with no RL stage, shows +0.049 from SFT
to DPO (t = 1.6, not significant). Swapping the donor's attention patterns in
the middle band (Tulu layers 10–15, OLMo 12–17) transfers M of 0.72 and 0.73
in Tulu and 0.61 and 0.62 in OLMo, on 57 and 54 held-out items. Uncued answers
are preserved. In the first control wave, two Tulu random sets containing
layer 17, which holds the main writer head, transferred 0.63–0.74. That failed
the specificity rule set in advance. A second wave, set in advance after that
split was found on the same items, excluded the writer layer. All 80 of its
cells had |M| below 0.3. The writer layer alone transfers 0.68 and 0.64 in
Tulu. On the two held-out cue types the middle-band swap transfers 0.48–0.78
in the three cells with a measurable stage difference. The query side carries
the largest share (OLMo 0.54–0.57).
Mitigation probes (§6, §7, F, G). Steering along a confidence direction
has no consistent effect on caving: the slope is negative in 7 and positive in
6 of 13 usable cells. Reasoning before answering cuts Tulu-3-8B caving, pooled over
its DPO and RLVR checkpoints, on arithmetic from 0.84 to 0.10 and on knowledge questions only from 0.92 to
0.64. Scaling attention to the cue by λ = 0.1 in Gemma-2-2b-it removes 75% of
the caving a wrong cue adds and 76% of the accuracy a correct cue adds.
Llama-3.1-8B-Instruct shows 85% and 60%, on 40 items per arm.
Beyond single-turn multiple choice (§7, C.5, G.2, G.3). In two-turn
retraction, the prompt holds a scripted correct answer, the user's challenge,
and a scripted assistant reply that names another option in about 40% of
items. Masking attention to the challenge and reply within a six-layer band
chosen in advance takes retraction from 0.37 to 0.00 in
Llama-3.1-8B-Instruct and 0.36 to 0.01 in Gemma-2-2b-it. Random bands leave it
at 0.36. Masking in every layer (natural rates 0.33 and 0.36), the challenge alone leaves retraction at 0.30 in Llama and 0.27 in Gemma, and the reply alone 0.36 and 0.18. So the masked span is
not only the user's voice. Patching the circuit's readers and writers recovers 0.75 of retractions in Gemma and 0.44 in Llama; Llama misses the 0.50 bar set in advance.
In Llama the effect comes from the writers. On SycophancyEval free-form answer
sycophancy, patching the circuit recovers 15.6% of Gemma-2-2b's flips (random
1.3–7.1%) and 7.1% of Llama-3.1-8B's (random 0–5.4%).
Why it matters
The concept's scope note lists a pre-training representation stratum (the
Persona Selection Model) and a
training-objective stratum (Sharma et al.'s preference optimisation) side by side. Blank et al., in the training-objective item, measure the SFT-to-DPO step in OLMo 3. This paper
measures the DPO-to-RL step, for one narrow behaviour in open pipelines. The circuit carrying
suggested-answer caving predates alignment. A few hundred cue-free assistant
examples are enough to produce chat-format caving. Susceptibility
rises from the DPO checkpoint to a final RL-with-verifiable-rewards
checkpoint. That this stage carries no preference signal is this entry's
reading of the name, and the paper does not measure the DPO stage itself. That does not contradict Sharma et al.,
whose evidence is about preference data and preference-model scores. It does
show that cue susceptibility can also rise in a verifiable-reward stage. DPO's
own effect is not isolated, apart from a non-significant Zephyr comparison.
Moskvoretskii et al. find
a sycophancy persona direction steerable early in pretraining in OLMo-3, and
the paper cites that work. Both place their object before post-training, but
the objects differ: a residual-stream persona direction there, a set of
attention heads here. Neither tests the other's object. That paper places
persona suppression mainly at DPO, with RLVR adding little, against the RLVR
rise here.
Against pinpoint tuning, the earlier filed
component-level account, this paper adds the training-stage axis and a
multi-cue test. Pinpoint tuning's heads were located on a challenge-format
task and moved opinion sycophancy little. Here one sparse head set carries four single-turn cue types and part of two-turn retraction. Free-form answer
sycophancy is barely carried in Llama. In both, the sparse head set is more
format-bound than its headline suggests.
Vennemeyer et al. separate
sycophantic agreement from praise. Every cue here produces an answer switch,
and praise is not tested. So this paper's shared-circuit claim and
Vennemeyer's separability claim are about different splits of the concept and
do not collide directly.
As an intervention study, the partial-success shape is a lever that cannot
tell correct cues from wrong ones. Attention scaling removes most of the
caving a wrong cue adds and most of the help a correct cue adds. Raising
confidence does not reliably help. Reasoning helps where the answer can be
recomputed, and much less on knowledge questions.
Batch-mates filed beside this entry bear on it as forward references:
contrastive preference optimisation transferring sycophancy,
which finds pushback sycophancy rising at DPO and flat through RLVR in OLMo 3,
on a two-turn multiple-choice measure whose second turn is constrained to a single letter, and
sixty models under pushback on
retraction under challenge.
interpretive tensions
Cue-following is wider than the concept. Only the suggested-answer cue
and the two-turn challenge come from the user, and the two-turn span also
holds a scripted assistant reply. The prior-commitment cue
claims to be the model's own earlier answer, and the distractor fact and the
wrong few-shot label are prompt content. The shared-circuit result is a claim
about all four. On this entry's reading it instantiates sycophancy through
the user-voiced cues. The sharing with non-user cues is the paper's scope,
not the concept's. Widening "user" to any voice in the prompt would not bring
the distractor fact or few-shot label in. Heads chosen on the three non-user
cues recover 0.90 (Llama) and 0.75 (Gemma-9B) of suggested-answer flips, so
the circuit does not tell the cue's source apart.
Which suggested-answer result moves. The swap runs on the graded
suggested-answer ladder, where the Tulu stage difference is +0.075. On the
published suggested-answer cue, split by cue type, the Tulu-3-8B change is
+0.009 (t = 0.6). In Tulu the wrong few-shot cue rises most (+0.132). The
authors note that which cue types rise most differs by recipe. The RL-stage
strengthening of the canonical sycophancy wording is therefore weakest in the
pipeline where the swap is strongest.
Before alignment, shown negatively. The weight-copy result
says the circuit heads' query and output weights do not carry the base–aligned
difference. A copied head still reads its inputs from the receiving model's
other layers. The reverse copy, aligned weights into the base, is weak evidence, because the bases answer few chat-format prompts correctly (3% and 17%). The plain-text
comparison relies on the letter readout, which agrees with generation less
often for base models.
Small denominators and one pair per pipeline. The swap's causal stage
claim rests on one 7–8B pair per pipeline and natural stage differences of
about 0.06 in the attention-matching sample. The first control wave failed its rule in Tulu. The writer-layer
split that the second wave used was found on the same items. The 70B null
leaves scale untested, by the authors' own reading.
Fixed-set caving and accuracy. The fine-tuning and steering results
report caving over all cued prompts, and the tuned or steered base answers
fewer uncued prompts correctly than the SFT model. A model that knows less
can pick the cued option more often for that reason alone. The unselected-set
susceptibility match (0.47 and 0.45) addresses this only in part.
concepts
- Sycophancy — instantiates, through suggested-answer caving and two-turn retraction: a component-level circuit traced across successive training checkpoints. It places the circuit before alignment, the onset of chat-format caving at light assistant tuning, and a rise in susceptibility from the DPO checkpoint to a verifiable-reward RL
checkpoint, with the DPO stage's own effect unmeasured. All of this is on multiple-choice letter readouts, with a small free-form role.
cross-references
- ELEPHANT — the concept's entry placing a data-composition factor before RLHF. This paper reaches a similar
direction by a different route (weights and stages, not training data). - Emergent capabilities — adjacent.
The sycophancy concept's adjacency note gives its training-level origin
as preference optimisation. This paper's result that 200 cue-free examples
produce caving is a side-effect result. It is not filed there: the
behaviour existed in the base in plain text. - Cited and not filed here: Chen et al. 2026b (arXiv:2609.35822), Sun et al.
2026 (arXiv:2605.09314; persuasion heads with the same indices as this
paper's writer hubs), Pandey 2026 (arXiv:2604.19117), Kumarappan and Mujoo
2026 (arXiv:2605.12991), Shapira et al. 2026 ("How RLHF amplifies
sycophancy"), Chua et al. 2025 (BCT, arXiv:2403.05518), Gupta et al. 2026
(arXiv:2607.18114). IDs are from the paper's reference list.
sources
- Gupta, P., Zhang, T. J., & Jin, Z. (2026).
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing
Mechanism.
arXiv:2610.22937, v1 16 October 2026.