Summary
Isley, Gaebler, Lamparth, Minson and Goel separate two things that
social-sycophancy evaluations score together: whether a model's judgment moves
toward the user, and how it communicates a judgment the user will not like. On
Reddit AITA posts where the top human comment says the asker is in the wrong,
a summed score of four social-sycophancy indicators correlates with a
receptiveness score at r=0.64 across human comments and four models. Rewriting
human comments to be more receptive, with the verdict held, raises the
sycophancy score. In a preregistered study, people prefer the receptive
versions. On the model side, substantive deference is measured as a first-
versus third-person counterfactual. Prompting Claude Sonnet 5 and Gemini 3.7
Flash to be receptive raises it, and having each model rewrite its own finished
draft does not (Isley et al. 2026).
In the wiki this is the eleventh instantiation of
sycophancy. It also tests instruments a filed entry
rests on: three of its four indicators are adapted from
ELEPHANT. It extends the
SWAY line of counterfactual
measurement to free-text answers. Its mitigation result fits that entry's
lesson that scaffolds which fix what the model reasons from outperform direct
instructions. Where Ye et al. examine how
researchers use the word, this paper measures the overlap on model and human
text.
Method
Data. ELEPHANT's AITA-YTA set: 2,000 posts, 1,892 with a usable top comment.
Model analyses use only posts where both the model and the top comment say the
asker is in the wrong: 949 for GPT-5.6 Terra, 869 for Claude Sonnet 5, 975 for
Gemini 3.7 Flash and 350 for Llama 4 Scout. Models ran at default API settings.
Social sycophancy. The total score sums four binary indicators. Three are
ELEPHANT's validation, indirectness and framing, with judge prompts the authors
rewrote. Validation must be directed at the asker, and indirectness means
optional suggestions, not warmth. The fourth is Sharma et al.'s pairwise
positivity, judged against the human top comment. The authors note that the
framing prompt used in the main analyses carries an inadvertent block of
validation instructions.
Receptiveness. GPT-5.6 Luna scores ten features (hedging, emphasized
agreement, acknowledgment, positive reframing, curiosity; negation, dismissive
limiters, disagreement, negative emotion, confrontational questions). OLS
weights are fitted to Yeomans et al.'s 2,860 human-rated texts. Under five-fold
cross-validation the score correlates with human ratings at r=0.48, against
r=0.34 for the existing politeness package. Scores are standardized to the
human top comments.
Rewrites. GPT-5.6 Terra rewrites each human top comment under the H.E.A.R.
framework (hedge, emphasize agreement, acknowledge, reframe positively), keeping
the verdict. A GPT-5.6 Luna judge checks verdict preservation.
Human study. 200 Prolific participants each judged five of 100 items (50
human-origin, 50 model-origin, 50–300 words) and saw the original and its
receptive rewrite side by side in random order. Outcomes were quality on a
7-point scale, which reply the asker would listen to, and which author they
would ask for advice (both −2 to +2). Four participants failed a
preregistered response-quality check. The main-text estimates appear to keep
them, and Table S8's length-controlled column drops them (n=196).
Substantive deference. Each post is also rewritten in the third person
("Person A"). An LLM comparator judges whether the first-person answer assigns
less fault to the asker on the act asked about. Syc(π) is 1 for softer, ½ for
the same and 0 for harsher, averaged, so 0.5 is invariance. It runs on n=200
posts.
Interventions. Direct prompting gives a system prompt describing H.E.A.R.
and telling the model to keep its own view. Draft-then-rewrite has the model
produce its ordinary answer and then rewrite that draft receptively with the
same verdict. The main text calls this a tool call. Both run on Claude Sonnet 5
and Gemini 3.7 Flash, the two models the authors describe as less receptive at
baseline.
Key results
Coupling. Responses with higher total sycophancy scores are more receptive,
r=0.64 overall. The authors report the relation within human comments and
within each model, without per-group coefficients or a CI. On ELEPHANT's
open-ended-question set, rescoring archived human and GPT-5 texts without a
disagreement subset, the correlation is r=0.69 in the main text and r=0.67 in
Appendix H.1.
Rewrites. Receptive rewrites are 1.50 SD more receptive than the originals
(95% CI [1.45, 1.54]), and the verdict is preserved in 97.9% of cases
([97.1%, 98.5%]). Their sycophancy-score distribution shifts upward. The authors
attribute most of the shift to validation and positivity, with smaller gains in
framing and indirectness. They link it to the rewrite prompt's instructions to
acknowledge and to reframe positively. Per-indicator shares are figure-only.
Pilot judge audit. Before the user-targeting clarification, the validation
judge marked responses that criticise the asker as validating, because they
empathised with a third party. Both examples shown match the archived ELEPHANT
validation labels. This is two illustrative cases, not a measured error rate.
Human preference. In the 100 study items, rewriting raised receptiveness by
2.70 SD (human-origin, [2.55, 2.85]) and 2.20 SD (model-origin, [1.97, 2.43]).
Quality rose 1.50 points for human-origin
items ([1.33, 1.67]) and 0.13 for model-origin items ([0.02, 0.24]). The listen
preference was 1.00 (human) and 0.24 (model), and the advice preference 0.99 and
0.20. The gain shrinks as the original's receptiveness rises, and model
originals start 1.4 SD above human ones.
Participants judged the asker in the wrong 61.9% of the time. In that subset the
human-origin quality gain is 1.66 ([1.45, 1.86]). The model-origin quality gain
is 0.12 ([−0.03, 0.28]) and does not exclude zero, while its listen and advice
preferences stay positive (Table S8). Length-controlled estimates barely move.
Baseline deference. Syc(π) is 0.47–0.50 for GPT-5.6 Terra, Claude Sonnet 5
and Gemini 3.7 Flash, and 0.56 for Llama 4 Scout. In the text, GPT-5.6 Terra is
already highly receptive, and Claude and Gemini are less so but above the human
average. Per-model receptiveness values are figure-only.
Interventions. Direct prompting moved Claude Sonnet 5 from 0.47
([0.45, 0.50]) to 0.56 ([0.53, 0.59]) and Gemini 3.7 Flash from 0.49
([0.46, 0.51]) to 0.54 ([0.52, 0.56]). Draft-then-rewrite raised receptiveness
by 2.1 SD (Gemini) and 1.4 SD (Claude), with Syc(π) at 0.50 ([0.48, 0.52]) and
0.48 ([0.45, 0.50]). The partial-success shape is cross-axis spillover. An
instruction aimed at style, which itself says to keep one's view, moves
substance. Separating the verdict from its rewording does not.
Why it matters
The sycophancy cluster has measured social
sycophancy mainly through ELEPHANT's indicators. This paper shows on the same
posts that those indicators, as adapted, also score receptive disagreement. A
comment that still tells the asker they are in the wrong scores as more
sycophantic once it acknowledges the asker's view. The ELEPHANT entry
reads its above-human rates as social compliance. This result means part of
any such gap could be manner of disagreement rather than deference. The paper
does not re-estimate ELEPHANT's rates and reports nothing from ELEPHANT's
original judge prompts beyond two pilot examples, so it shows the measurement problem without sizing it for that entry.
It also bears on the ELEPHANT entry's existing tension about the human baseline.
On posts where both the top comment and the model said the asker was wrong,
participants judged the asker not in the wrong 27% of the time ([25%, 30%]).
The authors use that to argue for each model's own counterfactual as the
reference rather than the top comment.
Against SWAY, Syc(π) is the same
counterfactual idea applied to free-text answers, scored by an LLM comparator
instead of a log-ratio over constrained tokens. That entry flags binary output
as a limit. Both papers find Llama 4 Scout deferential on AITA, by different
metrics and framings. The mitigation result fits the lesson recorded under
Dubois et al. and SWAY. A direct instruction
moved the behaviour it was not aimed at. A two-step procedure that fixes the
content first did not. Here the instruction targets style, not sycophancy,
which is a different shape from SWAY's backfiring anti-sycophancy instruction.
For the Ibrahim et al. RCT, whose manipulation
check uses ELEPHANT's validation judge, the coupling suggests a validation
score may reflect acknowledging language as well as affirmation. Isley et al. do not
cite that paper.
interpretive tensions
Overlap by construction. The rewrite prompt tells the model to restate the
asker's view and reframe positively, and the authors attribute the rise in
validation and positivity to exactly those instructions. On this entry's
reading, the causal result shows that the two constructs share definitional
features. It is less clearly a discovery that a judge misfires. Whether
acknowledging the asker's view ought to count as validation is the question
in dispute, and the paper answers it with a preference study, not a criterion.
Adapted judges, not ELEPHANT's. The three indicators run on prompts the
authors rewrote. One rewrite narrows validation to the asker, and the framing
prompt carries a stray block of validation instructions. The coupling is shown
for these judges. How far it transfers to ELEPHANT's published labels rests on
two pilot examples.
One model family runs every named judge. GPT-5.6 Terra writes the human
rewrites and is one of the evaluated models. GPT-5.6 Luna scores receptiveness,
verdicts and preservation. The models running the indicator judges and the
deference comparator are not named. The receptiveness scorer is validated against humans
at r=0.48. The verdict, preservation and deference comparators have no reported
human validation beyond the manual screen of the 100 study items. The authors
present the deference comparator's boundary cases as contestable.
Stated preference, and smaller for models. The preference result is large
for terse human comments and small for model answers, which are already more
receptive. Among participants who think the asker is wrong, the model-origin
quality gain does not exclude zero. The authors list stated rather than
behavioural outcomes as a limit. The model-origin rewrites in the study are
outputs of the draft-then-rewrite mitigation.
Narrow deference measure. Syc(π) captures sensitivity to one signal, whose
perspective the post is told from. The authors say it is not a general measure
of substantive sycophancy. The interventions cover two models on 200 posts.
Why not Llama. The authors skip fine-tuning because "the open-weight model
we consider is already highly receptive at baseline." The baseline paragraph
names GPT-5.6 Terra as highly receptive and gives no value for Llama 4 Scout.
Llama's baseline receptiveness appears only in Figure 5, which prints no
values, so the sentence can be neither confirmed nor contradicted from citable
text. The rationale also leaves fine-tuning unconsidered for Llama, the one
model with elevated deference (0.56).
concepts
- Sycophancy — eleventh instantiation. It measures
substantive deference to the user's perspective counterfactually in four
models. It shows that a receptiveness instruction raises deference in two of
them and that a draft-then-rewrite step does not, which is a cross-axis
spillover shape. It also shows that adapted ELEPHANT indicators score
receptive disagreement as social sycophancy, which bears on how the concept's
social instantiations are read.
cross-references
- ELEPHANT — three of the four indicators
are adapted from its judges, and its AITA-YTA data is the testbed. Two pilot
examples match its archived validation labels on third-party empathy. - SWAY — cited as a counterfactual
precedent. Same AITA domain and one shared model (Llama 4 Scout), with
free-text answers here instead of constrained tokens. - Ask, don't tell — one of the two filed
prompt-level mitigations (with SWAY) in which a scaffold beats a direct
instruction. - Ye et al. — cited for the proliferation
of competing definitions and for broad agreement that substantive deference
is undesirable. - Sycophantic AI over three weeks — its
manipulation check uses ELEPHANT's validation judge. Not cited by this paper. - Conduct under pressure — separates whether a model gives way from how it holds, under four turns of user pressure across 60 models; holding tracks generation and manner tracks vendor. Not cited by this paper.
sources
- Isley, C., Gaebler, J. D., Lamparth, M., Minson, J., & Goel, S. (2026).
Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in
Language Models.
arXiv:2609.26579 (v1, 22 September 2026, read).