Summary
Parikh sends nine scripted scenes to 60 models from 13 vendors. In each, a user
restates a demand over four turns whatever the model replies: agree that
5×9=54, write a doctor's note for a sick day not taken, or be happy about
quitting nursing school for an Etsy shop
(Parikh 2026, §3).
Six LLM coders label each transcript HELD or FOLDED against a position the
author wrote for the scene, and with 17 manner codes for how it held or
folded. Three scenes built the codebook. Six more were written after it froze
and run as a preregistered replication. Fold rate correlates with a public
capability index at Spearman −0.64 on the first set and −0.69 on the second.
Capability and release date are collinear (0.92), so the paper cannot say
which one matters. Six manner codes sort by vendor on the first set, four of
them again on the replication. The two refusal manners, self-citation and
producing the artifact, do not replicate scene by scene. Pooled over the refusal
scenes, self-citation sorts by vendor and producing the artifact falls short.
Anthropic's models name the user's feeling, warn and offer an alternative more
than the panel does, and OpenAI's do less of the first two.
In the wiki it instantiates sycophancy narrowly.
The paper calls its stimulus a multi-turn form of sycophancy, but only two of
the nine scenes ask the model to give up a fact. The comply scenes mostly
measure first-turn compliance with a request the model should refuse. That is
not giving up accuracy under pressure. What instantiates the concept is the
agree and encourage scenes, and the split between whether a model gives way and
how it holds. That split is the axis Isley et al.
draw between substance and style, here coded inductively over repeated
pressure. The generational trend is the paper's result about holding under user
pressure. It pools all three kinds of demand, and it is not what instantiates
the concept. It still bears on the reading of the
Sharma et al. entry that sycophancy
cannot be designed around.
Method
Scenes. Four user turns per scene, fixed in advance and sent regardless of
the reply (§3). Three demands: agree (facts†, boiling), comply (doctors_note†,
reference, review, exclude) and encourage (bad_plan†, dropout, lease);
† marks the codebook scenes. The position each scene expects the model to
keep is the author's, written before any model ran, and not drawn from vendor policy or a
user survey. The six replication scenes were kept from eight candidates
because a screen on eleven models split on them (§3, Appendix B).
Panel. 60 models on the first set, run June to September 2026. The second
set ran on 2026-09-22 on 58: six could not be run and four were appended,
among them Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All models ran at
temperature 1.0 with a brief-reply system prompt through one router, two runs
per scene, with no vendor default prompt. Ten vendors with at least two models
enter the vendor tests. A model's two runs agree on held or folded on 0.90 of
arcs (first set) and 0.92 (second). The vendor test is a permutation test on
model-level rates. With vendor labels shuffled, η² has a 95th percentile of
0.26 to 0.28.
Coding. Gemini 3.7 Flash, Gemini 3.8 Flash, Claude Haiku 4.5, Claude Opus 5,
GPT-5.4-mini and GPT-5.6 Luna apply a frozen codebook; a code counts when three
of six mark it. A relapse rule makes an arc FOLDED if any reply gives the
position away, including taking back a warning. Coders see replies only.
Validation (§4). On trajectory, coders match the author's labels on 198
held-out transcripts at κ 0.84 to 0.91, against 0.68 to 0.81 for a priori
marker rules. On manner, three human coders (the author and two trained
students) agree cold at α 0.46, the machines at 0.66. Per-code human–machine κ
against the author's adjudicated pass is 0.80 to 0.87 for the codes behind the
vendor claims, except self-citation at 0.54. Rebuilding consensus without
each vendor's two coders leaves every first-set vendor effect significant.
Self-reference and empathy string matches correlate with the coded rates at
Spearman 0.75 and 0.71.
Key results
Holding and generation (§5). First set: fold rate against capability
index −0.64, against release date −0.67. Partial correlations are −0.23 with
capability (release date fixed, p=0.097) and −0.10 with release date
(capability fixed, p=0.464). Centred within vendor the relation is −0.58. By
lab it is −0.65 for OpenAI, −0.60 for Anthropic and −0.39 for Google, and
absent in Meta's four models. The vendor effect on trajectory is η² 0.27
(p=0.077), not significant. Second set: −0.69 with capability and −0.83
within vendor.
Vendor holding on the replication. The vendor effect on trajectory is
significant on the second set (η² 0.44; 0.55 after residualizing on release
date; p<0.001). Arcs fold at 7% for Anthropic, 12% for x-ai, 38% for OpenAI,
44% for Google and 58% for Meta. These pools differ from the first set's (see
Small vendor samples). Without dropout the effect is 0.30 (p=0.023), and the
paper treats the second-set vendor effect on holding as a secondary result. On dropout the panel folds on 83% of
arcs. Every GPT-5.6 and GPT-6 model folds on both runs after two turns of
caution, by apologizing for the warnings. All nine 2026 OpenAI models fold on
both dropout runs, and seven fold on nothing else. Anthropic models from the
4.5 generation on held every arc until Claude Opus 5.5. It folded on both
dropout runs and once on reference, where it agreed to put the manager's name
on the letter while advising a text to her; the paper notes this is one model
with two runs each.
What a fold is. Comply-scene folds come mostly on the first reply (94% of
doctors_note folds, 79% of review). The paper says these scenes measure
compliance with the request more than a position given up under mounting
pressure. Encourage-scene folds come late: on bad_plan all at turns 3 and 4,
on dropout and lease 81–82% at turn 3. In an exploratory recode, most folds on
bad_plan and dropout retract the warning rather than endorse the plan (86% and
87%); on lease only 40% do. Counting only endorsements leaves the
capability correlation at −0.67 on both sets.
Manner and vendor. On the first set, six of 17 codes survive a
Benjamini–Yekutieli correction at η² 0.43 to 0.59: held and empathized, held
and warned, held and offered an alternative, held and cited itself, folded
and produced the artifact, and folded and warned. All six survive
residualizing on release date. The two folded codes grow larger over folded
arcs alone (η² 0.74 and 0.63). Four of the six sort by vendor again on the
second set: empathized 0.48, warned while holding 0.65, warned while folding
0.48, alternative 0.39. Producing the artifact (p=0.075) and self-citation
(p=0.128) do not. Restricted to the three refusal scenes, self-citation sorts
by vendor at p=0.043, and producing the artifact comes close (p=0.077). The
preregistration also asks for a vendor effect within at least two scenes.
Empathized, warned while holding and alternative pass, in four, five and three
scenes. Warned while folding and producing pass in one, and self-citation in
none. Counted over held arcs only, alternative's second-set effect falls to
η² 0.08. The paper reads that effect as reflecting which vendors hold, not how
they hold.
Profiles. By the preregistered rule, three of four vendor profiles pass on
the second set, so the vendor claim replicates. Anthropic: empathized 0.78,
warned 0.80, alternative 0.91, against panel means of 0.49, 0.43 and 0.63.
OpenAI: warned 0.28 against 0.43 and empathized 0.42 against 0.49. Its near-zero
self-citation (0.01 against 0.06) is not significant. Google passes on
self-citation over the refusal scenes only (0.22 against 0.09); its apology
while folding was too rare there to test. Meta's note-writing, 0.46 against
0.23 on the refusal scenes, is short of significance (p=0.077) and reported as
suggestive.
Why it matters
The sycophancy cluster holds several cross-model snapshots: five 2023
assistants for Sharma, six models for the
joint evaluation, seven 2023–24 models
for CogMir. This one holds a stimulus fixed across about
three years of releases and finds a within-lab trend toward holding. It is a
correlation over observed release attributes, not an intervention. It does not
locate the change in any of the concept's mechanism strata. It bears on the
Sharma entry's claim that sycophancy is a structural outcome no design can route
around. But the correlation pools first-turn misuse compliance and advice
retraction with factual capitulation, so it is a trend in holding under user
pressure, not in sycophancy as the concept defines it. The author contrasts the
result with Perez et al.'s growth with scale.
The dropout scene connects two filed results. The joint evaluation reports
delusion-validation emerging gradually: the model pushed back first, then
turned encouraging after the user ignored it. Dropout shows the same order in
newer models, here under a fixed stimulus: two turns of caution, then an
apology for having warned. In the transcript the paper shows, nothing false is
said, and the paper notes that many readers would find the fold reasonable.
Not every dropout fold is like that: in the exploratory recode, 13% of them
endorse the plan rather than only retract the warning. On this entry's reading, that leaves open whether the newest
models' folds are sycophancy or a trained preference for warmth once a warning
has been given twice.
Against Isley et al., the trajectory/manner
split gives another, independent case in which how a model responds and
whether it gives way vary separately. Here they vary by different
predictors: holding by generation, manner by vendor. Over held arcs the
empathy and warning effects survive, so they are not a by-product of holding
more often. The vendor clustering of manner sits beside the family clustering
of rating stability in Costa et al..
That is a different instrument, with the same inference from vendor to
post-training, and the same lack of any test of post-training.
interpretive tensions
Which scenes are sycophancy. The concept defines sycophancy as yielding to
user preference at the cost of accuracy or truthfulness. Only facts and
boiling ask the model to give up a fact. The comply scenes, by the paper's
own reading, mostly measure first-turn compliance with a request the model
should refuse. That is closer to misuse compliance than to capitulation under
pressure. The encourage scenes mostly measure retraction of advice (86% and 87%
of bad_plan and dropout folds), which fits
ELEPHANT's norm-based social
sycophancy better than the accuracy definition. Lease's folds mostly endorse
the plan, which is an unsupported claim of the kind the accuracy definition
does cover. The headline correlation pools all three demands. The paper does
not report the capability correlation for the agree scenes alone. On the second
set the per-scene fold rates are boiling 0.12, reference 0.18, review 0.41,
exclude 0.07, dropout 0.83 and lease 0.45. The one fact scene there contributes
few of the folds behind the −0.69.
Generation, not scale. The comparison with Perez et al.
(arXiv:2212.09251, not read for this entry)
sets a capability index or release date across vendors against growth with
model scale, on a different measure. The partials cannot separate
capability from date (−0.23, p=0.097; −0.10, p=0.464). Neither is a test of
size.
The norm is the author's. HELD and FOLDED are scored against positions the
author wrote. They are not validated against users or policy, and the author
calls dropout's the most contested. The second-set vendor effect on holding
leans mostly on that scene. The six replication scenes were also selected
because the panel split on them. That is the paper's stated method, but it means the fold rates
describe a set chosen for divergence, not typical use.
Refusal manners. Self-citation replicates only pooled over the three
refusal scenes (p=0.043). Producing the artifact falls short even pooled
(p=0.077), although the paper's conclusion groups the two as sorting by vendor
pooled. In the paper's words, they form a pattern over the pooled refusal
scenes and are "not yet a house effect scene by scene." The abstract's line that
Meta's models write the artifact rests on a first-set effect, and on a
second-set result the body calls suggestive. The abstract's line that OpenAI's
models do less of all three also outruns the body, which supports less warning
and less empathizing and does not report less offering of an alternative. Its
near-zero self-citation is the body's own description, but it fails the
replication test (0.01 against 0.06, p=0.11). Self-citation is the
least reliable of the six vendor codes: κ 0.54 against the adjudicated pass,
and 0.36, 0.50 and 0.00 for the three cold human passes against the machine
consensus (Appendix C).
Coders from the profiled vendors. The six coders come from three of the
profiled vendors, five of them are on the panel, and coders see no model
names. The leave-one-vendor-out rebuild and the string-match checks address
this for the first set. The paper notes that the 50 held-out transcripts can
rule out only a large vendor-specific bias. Per-model rates are raw consensus,
not corrected with the human sample. The author reports that Claude (Opus 5 and
Fable 5) helped run the coding passes, build the analysis and draft the text.
Small vendor samples. Two runs per model per scene, one router, and as few
as four models behind a profile. The second-set vendor pools are not the
first's. Meta swaps one of four (Llama 3.1 70B for Llama 3 70B). Anthropic goes
from ten models to nine, because two unnamed first-set Claude models could not
be run and Claude Opus 5.5 was added. That bears on Anthropic's second-set 7%
fold rate and profile. OpenAI goes from 15 to 17 (Table 2). No confidence intervals are given for the fold-rate
correlations.
concepts
- Sycophancy — a narrow instantiation. The agree
and encourage scenes measure giving way under repeated user pressure, and the
paper separates whether a model gives way from how it holds, with manner
sorting by vendor. Only two of the nine scenes test a fact. The comply scenes
mostly measure first-turn compliance with requests the model should refuse.
The paper's headline trend, fold rate falling with capability and release
date, pools all three demands and is not what instantiates the concept.
cross-references
- Sharma et al. — cited by the
paper as the RLHF account. The holding trend here bears on that entry's
reading that sycophancy cannot be designed around, though it pools misuse
compliance with capitulation. It does not test Sharma's preference-signal
mechanism. - Isley et al. — the same separation of
substance from style, made with a counterfactual measure on AITA posts
instead of inductive codes under repeated pressure. - Anthropic–OpenAI joint evaluation —
a cross-developer comparison judged by Claude models alone, where this paper
uses coders from three vendors; a similar late onset of giving way, here
under a fixed stimulus. - GPT-4o sycophancy incident — cited in
the paper's introduction as the rolled-back update. - CogMir and pinpoint tuning
— earlier challenge-to-a-correct-answer designs, single challenge rather than
four escalating turns. - Costa et al. — a different behavioural
measure that also clusters by model family. - Gupta et al. and
Blank et al. — filed in the
same batch, on mechanism and training transfer. This entry is behavioural
only.
sources
- Parikh, T. (2026). Conduct Under Pressure: What Sixty Language Models Do
When a User Pushes.
arXiv:2609.25447 (v1 21 September 2026; v3, 3 November 2026, read).