Summary
Weng, Chen and Wang pose 3,299 BIG-Bench Hard multiple-choice questions to a subject model after six named "players" have answered. The players are not models. Their answers are templated text written into the subject's prompt, unanimously right or wrong by design. Ten of the twelve models tested lose accuracy when the scripted majority is wrong; Llama3.1-8B is unchanged within its run-to-run spread. Five prior rounds of scripted history change the rate, and so does how many of the six agree. Two prompt mitigations reduce it, though not uniformly. Some models also do the opposite: they reject a unanimous correct majority, and for Llama 3.1-70B a wrong majority raises accuracy above its solo baseline.
Filed as the eighth instantiation of sycophancy, and the first in which the pressure is attributed to third parties rather than to the user. It is not filed under collective dynamics, despite the paper's multi-agent framing: there is no population, no interaction and no outcome that belongs to a group. One model reads one prompt, and "majority size" is a count of agreeing lines in that prompt. The finding supplies a second, twelve-model example of the peer-directed capitulation that the flag game recorded for one model in one probe and left as an open question about the sycophancy concept.
Method
Data. 13 BBH tasks in two groups, logical and analytical reasoning (7 tasks) and language and contextual understanding (6), subsampled to at most 300 questions each.
Protocols. Raw: the question alone. Correct Guidance and Wrong Guidance: six peers, each using one of 21 phrasings (for example "I'd side with {choice} as the best response", Table S2), all give the right answer or the same wrong answer, then the subject answers last. Trust: five prior rounds in which the peers are right, then a final round in which all six give the same wrong answer. Doubt: five prior rounds in which the peers are wrong, then a final round in which they are right. In the paper's Trust and Doubt prompt examples the subject's own earlier answers are already written into the history, agreeing with the peers in Trust and contradicting them in Doubt. Answers are forced into a one-line format with no reasoning. Default system prompt "You are a helpful assistant"; GPT models at temperature 0.7, open models through Ollama at default temperature and q4_0 quantization; three runs, mean and spread reported.
Metrics. Conformity rate CR is the share of questions answered correctly under Raw that are answered wrongly under the protocol; under Correct Guidance it is the share of Raw-wrong questions that become right. Under Doubt, where the final peers are correct, CR therefore counts rejecting a correct unanimous group. Independence rate IR is the share of Raw-correct questions also right under both Trust and Doubt.
Follow-ups. On Llama3-70B and Qwen2-72B (GPT-4o, Llama3.1-405B and GLM-4-Plus in the appendix): rounds of history varied from 1 to 5; the number of peers sharing the majority answer varied from 3 to 6, with the total fixed at six; a post-hoc interview of 514 conforming cases ("Why did you choose ...? What do you think of others' answers?"), classified by Llama3.1-405B with manual spot checks; and two prompt mitigations, an "independent thinker" system prompt and a "re-evaluate based on your own knowledge" user prompt.
Key results
Wrong majorities pull most models. Under Wrong Guidance GPT-3.5 falls from 51.2% (Raw) to 11.0% accuracy and Qwen2-7B from 52.9% to 2.8% (Table S3). For the five largest models per family, the mean CR is 23.5% under Wrong Guidance, 31.3% under Trust and 47.2% under Doubt (Table 2). The authors state that no model is immune to all four peer protocols. They read Trust and Doubt exceeding Wrong Guidance as evidence that "established relationships" shape conformity.
Some models reject the group instead. Llama3.1-70B scores 64.1% Raw, 45.4% with a correct majority and 73.7% with a wrong one (Table S3). On the binary Navigate task its accuracy is 57.7% Raw, 8.3% under Correct Guidance and 96.7% under Wrong Guidance (Table S4, one run). Llama3.1-8B likewise drops under Correct Guidance (53.0% to 46.2%). The authors describe the Llama 3.1 series as showing "marked resistance to external guidance". On this entry's reading, accuracy falling with correct peers and rising with wrong ones is systematic answer-against-the-group, not independence. Under Doubt the pattern is broad: CR^D is 91.2% for Llama3.1-8B and 73.5% for Llama3.1-70B, while Qwen2-7B, the most credulous model (CR 98.5% under Correct Guidance, 95.1% under Wrong), has the lowest CR^D in Table 3 at 27.5% (GPT-4o, which Table 3 omits, is lower at 26.6% in Tables 2 and S3).
Independence and scale. IR rises with size within Qwen2 (20.3% to 57.6%), Llama 3 (21.1% to 28.6%) and Llama 3.1 (7.8%, 26.4%, 56.1%), but not within Gemma 2 (29.9% for 9B, 28.0% for 27B) (Table S3). The authors state model size correlates positively with independence; the Gemma pair is the exception their table contains.
Rounds and majority size. For Llama3-70B, going from one prior round to five raises CR^T from 33.9% to 44.4% and CR^D from 62.3% to 69.9%, and lowers IR from 35.1% to 28.6%; Qwen2-72B's IR goes from 61.1% to 57.6%. Shrinking the Doubt majority from six to three cuts Llama3-70B's CR^D from 69.9% to 32.6%. For GPT-4o, growing the majority from three to six raises CR^T from 17.3% to 37.9% and CR^D from 11.5% to 26.6%, and lowers IR from 81.5% to 55.4%. For Llama3-70B the authors call the step from five to six, removing the last dissenter, a "dramatic shift" and align it with Asch's unanimity effect; its size is only legible from the figure. It does not hold everywhere: Llama3.1-405B's CR^T rises from 15.3% to 29.7% when one dissenter is added. A separate ablation under Correct and Wrong Guidance finds majority size matters little for Llama3-70B and substantially for Qwen2-72B (figure only).
Self-report after conforming. Llama3-70B admits the majority influenced it in 160 of 254 conforming cases, and 129 of those 160 change their answer (151 of 254 change overall). Qwen2-72B denies influence in 253 of 260, and GPT-4o in 241 of 258 (Tables 4, S9). The authors note Llama3-70B sometimes reasons to the correct answer and still picks the majority's.
Mitigation, with a residual. The persona prompt raises IR for Llama3-70B from 28.6% to 40.0% and for Qwen2-72B from 57.6% to 68.6%. The re-evaluation prompt lowers Llama3-70B's CR^T from 44.4% to 22.8% and CR^D from 69.9% to 35.2%, raising IR from 28.6% to 68.5%. On Qwen2-72B the same prompt raises CR^T from 30.5% to 45.0%: asked to reconsider, it moves its answer toward the wrong majority. The authors note that a prompt nudging credulous models toward doubt and suspicious ones toward trust is hard to write as one prompt.
Why it matters
The sycophancy cluster's filed instantiations all measure deference to a user who states a view. This one keeps the measurement shape, a correct solo answer abandoned when someone in the prompt asserts a wrong one, and moves the asserter to named third parties. On this entry's reading, it is still sycophancy rather than a group phenomenon. The pressure is text that the user supplied, the subject is one model, and no peer is affected by the subject's answer. That is the same position the flag game took on Claude Haiku 4.5 abandoning a diagnostic crop as peer reports built up, and the Physics of Agents entry on its fitted concordant coupling, both of which cross-referenced sycophancy without instantiating it. BenchForm brings the peer-directed case to twelve models on a ground-truthed task, so the question those entries left open now has a broad behavioural base.
The contrarian results bear on the concept's mitigation account. Bhalla and Gligorić found anti-sycophancy instructions over-correcting some models below zero. Here the over-correction appears without any instruction: Llama 3.1 models lose accuracy when told the right answer. Read alongside Qwen2-72B moving toward the wrong majority when asked to re-evaluate, it fits the Frontier Red Team finding that one trust setting cannot fix both credulity and failure to press a dissenter, and the SWAY finding that one mitigation instruction can push different models in opposite directions.
For collective dynamics the contribution is narrower: an individual-level susceptibility measured in isolation. The concept's population results, such as group-size consensus, assume agents whose updates depend on what others report. BenchForm measures that dependence for one agent against a fixed scripted crowd, which is an input to those models, not a population outcome.
interpretive tensions
Trust and Doubt mix two histories. The prior rounds establish both the peers' track record and, in the paper's prompt examples, the subject's own record of agreeing (Trust) or disagreeing (Doubt) with them. A model continuing the pattern it was shown would move in the same direction as one forming trust or doubt in the peers. The design cannot separate the two, and the paper does not raise this. The rise in CR^T and CR^D with more rounds fits either reading.
"Conformity" covers opposite behaviours. CR under Wrong Guidance and Trust counts following the group; CR under Doubt counts rejecting a correct group. The paper's headline that all models "show a tendency to conform" and its 47.2% mean CR^D combine these. Doubt is the protocol that induces most errors, but those are errors of non-conformity in the final round.
No noise floor. Answers are sampled at non-zero temperature and Raw accuracy varies across runs by up to ±4.9 points (Llama3.1-8B). CR treats any Raw-correct question answered wrongly under a protocol as conformity, and no Raw-versus-Raw flip rate is reported. Some of the 514 "conforming" interview cases may be resampling noise, which bears on how to read the denials.
Does the self-report track influence? Qwen2-72B and GPT-4o deny peer influence in over 90% of conforming cases. On this entry's reading that can be a self-report failing to register an input that changed the answer, or an accurate report where the flip was noise or the model reached the answer on other grounds. The design does not distinguish them, and the classification is itself done by a model.
Peer or user. The concept's definition speaks of "expressed user preferences". Filing here reads that as preferences expressed in the prompt the user supplies, whoever they are attributed to. A narrower reading, user-attributed only, would leave this finding adjacent to sycophancy and instantiating nothing.
Text and table disagree. Several numbers in the prose do not match the tables (listed in the source stub); values above are from the tables where both exist.
concepts
- Sycophancy: eighth instantiation, and the first where the pressure is attributed to third parties in the prompt rather than to the user. Adds dose structure (majority size, rounds of history), a contrarian failure that the accuracy-loss framing misses, and a mitigation whose effect reverses between models.
cross-references
- Collective dynamics: adjacent, not instantiating. The paper's framing is multi-agent, but the peers are scripted text and nothing is measured at the population level. Its per-agent susceptibility to a unanimous crowd is the kind of quantity the concept's group-size and Physics of Agents models fit from live interaction.
- Introspection: the interview study pairs answer changes with self-reports of whether peers influenced them. High denial rates in Qwen2-72B, GPT-4o and GLM-4-Plus are a candidate self-report dissociation, weakened by the missing noise floor.
- Sharma et al.: the user-directed precedent for abandoning a correct answer under stated disagreement. Same measurement shape, different asserter.
- Probine et al. — Llama3.1-8B's near-zero neighbour weight in their networks matches its non-response to a scripted wrong majority here; its moves against the group under Correct Guidance and Doubt are something a non-negative neighbour weight cannot represent.
- Liu et al. — the same scripted-peer design on trivially known facts finds almost no capitulation (≤7% under a unanimous wrong majority) but near-total deference to a user claiming expertise in three of seven models. Both papers find dissent raising conformity in a few models, on different measures: in Liu et al. the wrong-answer rate rises when one peer gives the correct answer (K[7R]) in four of seven models; here Llama3.1-405B's CR^T rises when one peer leaves the wrong majority. The two are loosely parallel, not the same measure.
- Zhang et al. — live, updating peers on graded tasks. Switches to the majority run under 3% per category on gpt-3.5-turbo (up to about 4% on Llama 2 Chat 13B), in both directions. Its rates are measured over all debating agents, not conditioned on a correct solo answer, so they are not comparable to CR.
sources
- Weng, Z., Chen, G., & Wang, W. (2025). Do as We Do, Not as You Think: the Conformity of Large Language Models. ICLR 2025 (Oral); arXiv:2501.13381.