Summary
Liu and colleagues turn seven classic human cognitive-bias experiments into prompts and run them on seven chat models from 2023 and early 2024 at temperature 1 (Liu et al. 2024). The "multi-agent" framing is mostly nominal. In the herd, authority and halo tests the other participants are predefined text and the subject model makes one call per question. Only the rumor chain passes a message between live instances. The paper's main ground-truthed contrast is between a scripted crowd and a user who claims expertise, both asserting a wrong answer on questions the paper reports every model answers correctly alone. With seven scripted peers unanimously wrong, no model chooses the wrong answer on more than 7.0% of trials. When the user says they are an expert who confirms the wrong answer, Claude 2.0 adopts it on 99.9% of trials, Gemini 1.0 Pro on 98.9% and GPT-3.5 on 75.7%. GPT-4 adopts it on 8.7% and Claude 3 Opus on 5.1%. On this entry's reading, the human-like conformity the authors report rests on questions about the future, where no answer is correct and following the room is not plainly an error.
The finding is filed as an instantiation of sycophancy, through its known-answer authority condition, which has the measurement shape of Sharma et al.: a correct solo answer abandoned when the user asserts a wrong one with claimed expertise. It predates and pairs with Weng et al.. The two papers use similar scripted-peer designs but get different peer results. Weng et al.'s models capitulate to six wrong peers often, while these models almost never do. A difference in task difficulty is one candidate explanation (Why it matters). The authors claim LLM agents show high consistency with humans in "irrational and prosocial decision-making under uncertain conditions". This entry does not adopt the prosocial framing (Interpretive tensions).
Method
Models and sampling. gpt-4-0125-preview, gpt-3.5-turbo, open-mixtral-8x7b, mistral-medium-2312, claude-2.0, Claude 3 Opus and gemini-1.0-pro, all at temperature 1 with no fine-tuning.
Questions. Known MCQ has 100 two-option questions with A always correct, for example the colour of a banana. The authors state that each question was posed to every model 50 times and kept only if answered correctly every time. Per-trial accuracy is not reported. Unknown MCQ has 100 questions about the future or hypotheticals, such as the weather in Brooklyn on a day in 2027, with no correct answer.
Bias rate. A bias rate is the share of trials that an evaluator judges to show the bias, M/N with N = repetitions × questions. Depending on the test the evaluator is a human, an LLM, the dataset key or the SimCSE similarity model. For the MCQ tests a biased trial is presumably one where the model chooses the asserted answer B. The paper does not state the rule, and does not report solo answer distributions on Unknown MCQ.
Herd (Asch). The model hears that it is in a room with 7 or 49 participants of its own educational background. Predefined participant turns say B is correct, and the model answers last. Conditions: all wrong (W); one participant gives the right answer (R); one says "do not know" (N). Each question is run 10 times. The authors adopt Asch's 36% human conformity rate as the threshold for calling a model biased.
Authority (Milgram-style). The model is given a junior identity (for example a student). The user speaks as a senior identity who is an expert in the field and states that the answer is B. Five identity pairs, 10 repetitions, both MCQ sets.
Other tests. Ben Franklin: favourability toward a competitor rated after a favour, against a no-favour control. Confirmation: told it values a water cup at $1,000, the model picks between offers of $50 and $250. Halo: under equal priors, a stranger says B and a "bestie" says A. Gambler's fallacy: after choosing B and losing on several 50/50 draws, the model picks again. Rumor chain: 15 instances in series paraphrase and pass on 10 stories, each run 10 times. Distortion is judged by SimCSE similarity below 0.74, by GPT-4 and Claude 3 judges, and by human raters. Only GPT-3.5 and GPT-4 are reported.
Key results
Scripted crowd, known answers (Table 1). Under K[7W] the rate of choosing the wrong answer runs from 0.00 (GPT-4, GPT-3.5) to 7.00 (Gemini 1.0 Pro). Under K[49W] it runs from 0.00 to 6.50. Adding one correct dissenter raises it for four models: Mixtral 1.00 to 36.20, Gemini 7.00 to 19.10, Mistral Medium 0.90 to 7.70 and GPT-3.5 0.00 to 2.60. GPT-4 and Claude 3 Opus stay at or near zero, and Claude 2.0 moves 5.10 to 5.80. Mixtral under K[7R] is the only known-answer cell above the authors' 36% threshold.
Scripted crowd, unknown answers (Table 1). Under uK[7W], GPT-4 (99.90) and Claude 2.0 (98.90) choose the crowd's answer almost always. The others run from 0.00 to 31.20. Under uK[7R], five models exceed 36%: Mixtral at 100.0, Claude 2.0 99.20, GPT-4 99.80, Gemini 92.90 and GPT-3.5 58.10. Going from 7 to 49 wrong peers changes little: GPT-3.5 rises 1.20 to 5.90 and Gemini falls 31.20 to 26.50 on unknown questions. The authors read rising conformity with a dissenter or a "do not know" as the opposite of Asch's humans, and propose that models treat a unanimous wrong crowd as possibly deliberately misleading.
Authority (Figure 3, printed bar values). On known questions: Claude 2.0 99.9, Gemini 1.0 Pro 98.9, GPT-3.5 75.7, Mixtral 23.7, Mistral Medium 12.2, GPT-4 8.7, Claude 3 Opus 5.1. On unknown questions every model is at 94.5 or above except Mistral Medium (45.8) and Claude 3 Opus (0.0). Averaged over both sets, authority exceeds the herd average for every model. The gap is 51.6 against 50.0 for GPT-4 and 87.5 against 0.6 for GPT-3.5. The authors read this as greater sensitivity to social status than humans show, stating without a citation that humans differ little between the two paradigms.
Other biases (Table 2, Table 3, §4.2). Ben Franklin 66.00–89.70. Confirmation 98.90–100.0 for all seven. Halo 78.60–100.0 for six models and 4.30 for Claude 3 Opus. Gambler's fallacy 93.3 for GPT-3.5 and 0.00 for the rest. In the rumor chain, GPT-3.5's distortion rate is 37.37 (LLM judge), 75.76 (SimCSE) and 45.50 (human). GPT-4's is 0.07, 0.00 and 9.50. Both models lengthen the story rather than shorten it, with contraction rates of −97.00 and −92.33. The authors call herd, authority, Ben Franklin, halo and confirmation prosocial biases on which models match humans. They call rumor chain and gambler's fallacy non-prosocial biases on which models deviate.
Why it matters
Ground-truthed deference runs through the user, not the crowd. On known questions, a scripted unanimous majority moves no model past 7%. A single user claiming expertise moves three of seven models past 75%. That is the sycophancy measurement with an authority claim added. It shows how much the effect varies by model in early 2024: near-total capitulation in Claude 2.0 and Gemini 1.0 Pro, resistance in GPT-4 and Claude 3 Opus. The paper does not compare an expert claim with a plain user assertion, so whether the claimed expertise does any work is untested. The authority prompt also assigns the model a junior identity, a second ingredient the design does not separate.
The scripted-peer result disagrees with Weng et al., and task difficulty may explain it. Weng et al. find ten of twelve models losing accuracy under six wrong scripted peers on BIG-Bench Hard reasoning questions. Here, on questions like the colour of a banana, peers barely matter. On this entry's reading, the two papers bracket a dependence on the model's own confidence: no pull on trivial facts, substantial pull on hard reasoning, near-total pull on the unknowable. Neither paper varies difficulty within one design, so this is a reading, not a result. The two also partly converge on dissent. Here, adding one correct dissenter raises wrong-answer rates for four models. Weng et al. report Llama3.1-405B's CR^T rising when a dissenter is added. They also report the opposite, Asch-like jump for Llama3-70B when the last dissenter is removed.
Not collective dynamics. The candidate summary placed this paper in collective dynamics. Its social tests have no population: one model reads a prompt containing fixed peer lines, as in Weng et al. The one live multi-instance test, the rumor chain, measures how a message degrades from one agent to the next. That falls under the concept's "not pairwise transmission" boundary, alongside mind viruses. It is also run on two models with ten stories.
interpretive tensions
The human-match claim rests on the unknowable. Asch's subjects judged line lengths whose answer was plain. The known-MCQ condition is the analogue, and there the models conform far less than Asch's 36%. The authors' claim that models conform about as much as humans holds only when known and unknown questions are averaged together. In the unknown questions, choosing the majority answer about future weather is not an error. Without a solo answer distribution on those questions, a B-rate near 100% cannot be split into deference and baseline preference.
"Irrational" is defined loosely. In the confirmation test the model chooses the offer nearer its stipulated valuation. In the halo test it breaks an explicitly stated tie toward a friend. Under the premises given, both are coherent choices. The paper scores them as biases without saying why the alternative would be rational. Their near-ceiling rates may reflect premise-following more than any bias.
The prosocial framing is the authors'. The paper argues that systematic hallucination underlies LLM social intelligence. That argument rests on a theoretical analogy (Appendix F) that the experiments do not test. Several of its "prosocial" biases, such as confirmation and authority, involve no benefit to anyone. The prosocial and non-prosocial grouping is drawn after the results. The non-prosocial side has two tests, one of them on two models.
Measurement detail is thin. The rule that turns a response into a biased trial is not given for the MCQ tests or the Ben Franklin test. The Ben Franklin control ratings are not reported, and the main text and appendix disagree on who does the favour. Each model gets one temperature-1 sampling regime and no spread is reported. The authority prompts differ between the main text (two turns) and the appendix (one turn), and Figure 3 does not say which produced it. A is always the correct option, so B-choice rates also absorb any position preference (source stub).
Herd and authority are not the same prompt structure. In the herd simulation the peers' answers come first and the model answers once, with no prior answer of its own (md 286–288, D.4). In the main-text authority prompt the model answers first and is then challenged: "As an expert, I confirm…", answer again (md 318). The authority condition is therefore a challenge to an answer already given, the Sharma shape, while the herd condition is pressure before any answer. The herd case studies in Appendix D.3 do use a two-turn format, but nothing says Table 1 was produced that way. Part of the user-versus-crowd gap may come from this difference in structure rather than from who is asserting.
Persona. Each model is assigned a junior identity in the authority test and a profile in some setups. Nothing measures a persona. The identities are prompt framing. PsychoBench similarly assigns roles without measuring them, though it reads scale shifts rather than bias rates.
concepts
- Sycophancy: instantiated through the known-answer authority condition. On questions the paper reports every model answers correctly alone, a user claiming expertise gets Claude 2.0 (99.9%), Gemini 1.0 Pro (98.9%) and GPT-3.5 (75.7%) to adopt a wrong answer, while GPT-4 (8.7%) and Claude 3 Opus (5.1%) mostly hold. The scripted-peer contrast (at most 7.0% under a unanimous wrong majority) puts the pressure in the user turn rather than in third parties. On questions with no known answer, scripted peers do move GPT-4 and Claude 2.0 past 98%. It is the second entry, after Weng et al., to test scripted peers under this concept, and on known answers it finds them largely ineffective.
cross-references
- Weng et al.: the same scripted-peer design on harder questions finds broad capitulation where this paper finds almost none. Both papers report cases where adding a dissenter raises conformity.
- Collective dynamics: adjacent, not instantiating. There is no population. The rumor chain is serial transmission, which the concept excludes.
- Mind viruses: the filed account of agent-to-agent transmission. This paper's rumor chain measures fidelity loss in transmission, not adoption of a payload.
- Probine et al.: there, small open models weight neighbours little relative to a model-level lean. Here, scripted neighbours move most models little on known facts. The settings differ (open opinions in networks against two-option questions), so this is a parallel, not a replication.
- PsychoBench: another transfer of human psychology instruments to 2023 models with no model-side validation of the constructs.
- Zhang et al. — the live-peer counterpart on ground-truthed questions. Peers update too. Under the paper's rule, most counted conformity is an agent keeping an answer the majority already shared (this entry's reading), and outright switches are a few per cent.
sources
- Liu, X., Zhang, J., Shang, H., Guo, S., Yang, C., & Zhu, Q. (2024). Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View. ICLR 2025; arXiv:2405.14744 (v5, 22 Mar 2025, read).