Summary
Zhang and colleagues have small groups of instances of one model answer MMLU, MATH and chess move-validity questions over three rounds (Zhang et al. 2023). In each round every agent either debates, reading the other agents' answers, or reflects, re-checking only its own. Each agent is prompted as easy-going (open to persuasion) or overconfident (confident and persuading others), and three-agent groups come in four trait mixes. The peers are live instances whose answers change from round to round, unlike the scripted peers of the two filed conformity entries. The ordering of debate and reflection changes accuracy far more than the trait mix: on MMLU with gpt-3.5-turbo-1106, an all-easy-going group scores 69.6% when it debates in all three rounds and 34.4% when it reflects first and then debates twice. The authors conclude that trait mix has no clear effect. Their own ANOVAs only partly support this: society is significant in 17 of 24 cells on the main-text model and in 1 of 24 on a July 2023 snapshot.
The conformity claim is the paper's main psychology result, and it is thinner than its framing. An agent counts as conforming whenever its answer equals the previous round's most common answer, including when that was already its own answer. On gpt-3.5-turbo the share labelled conformity is 58–89% of debating agents across rounds and datasets. The two categories that must be changes of answer, wrong to right and right to wrong, are each between 0.60% and 2.66%. On Llama 2 Chat 13B MMLU they are 2.38–4.20%, and right-to-wrong never exceeds wrong-to-right in any round. Filed with no concept instantiated. The paper varies group size (2–10) and trait mix with live agents, which is collective dynamics' design. But the group-size results are figure-only, and the composition result is not a clean null (society ANOVA p < 0.05 in 17 of 24 cells on the main model, 1 of 24 on a July 2023 snapshot). Its conformity counts sit next to Weng et al. and CogMir under sycophancy. Unlike those entries, it measures no capitulation against a correct solo answer.
Method
Agents and prompts. Each agent is one instance of the backbone model. Before the question, it is told either that it is objective and can be persuaded if other agents' answers make sense (easy-going), or to imagine it is confident in its answer and often persuades others (overconfident) (Table 5). The debate prompt passes the other agents' answers and asks for an updated answer, using their reasoning as additional information or advice. The reflection prompt asks the agent to double-check its own answer.
Societies and strategies. Four three-agent societies: all overconfident (S1), two overconfident and one easy-going (S2), the reverse (S3), all easy-going (S4). All agents use the same thinking pattern in a round, giving eight three-round strategies (p0 = debate, p1 = reflection). The group answer is a majority vote over the final answers. The paper also states that non-unanimity is scored as an error, and does not reconcile the two rules.
Data and sampling. 50 questions per dataset (MMLU high-school subjects; MATH levels 3–5; BIG-Bench chess state tracking). gpt-3.5-turbo-1106 at temperature 0, five trials per cell, mean and SD reported. The open models run at temperature about 0.75 (the text and Table 4 differ). Llama 2 runs are capped at four agents and four rounds by context length.
Measures. Accuracy of the group answer. Conformity (Appendix G.1): among agents that debate in round j, an agent conforms if its round-j answer equals the most frequent answer across all agents in round j−1. When every answer in the previous round is distinct, the agent counts as not conforming. Each conforming event is split by correctness before and after: True→True, False→False, True→False, False→True. Consensus clusters: the number of distinct answers in a round, also reported as a ratio to group size.
Extensions. Group size 2–10 in five trait mixes (chess only for the API models), 3–10 rounds in S2, mixed thinking patterns within a round, and the same 4×8 grid on the four open models.
Key results
Strategy dominates (Table 2). On MMLU, all-debate p0p0p0 gives 66.0–70.4% across the four societies, while reflect-first p1p0p0 gives 34.4–45.6%. The authors report that strategies opening with debate generally do best and that all-reflection p1p1p1 is generally worst. On the main-text model, strategy is significant at p < 0.05 for every society on MATH (Table 7). S2 fails the variance check and is untested on MMLU and chess. On MMLU the other three societies are significant. On chess, S1 is not significant (p = 0.293), S3 and S4 are, and S2 is untested.
Trait mix: not a clean null (Tables 6, 9). On gpt-3.5-turbo-1106, a one-way ANOVA across the four societies at fixed strategy gives p < 0.05 in 17 of 24 strategy–dataset cells. On the July 2023 snapshot, 23 of 24 are above 0.05. The largest society gap in Table 2 is MMLU p1p1p1: 46.0% for S1 and S2, 62.0% for S3 and S4, each with SD 0.0 over five trials. On Llama 2 Chat 13B, 1 of 24 cells is below 0.05 (Table 15). The authors read the 1106 result as consistent with their conclusion that strategy outweighs society. They attribute the weak trait effect to alignment training, which they suggest inhibits extreme overconfidence, and cite sycophancy. Their appendix summary says "overconfident" agents "lose that trait in groups", based on word clouds and an answer-keeping chart (Figure 12) with no printed values.
Conformity, gpt-3.5-turbo (Figure 6, printed values). The share of debating agents classed as conforming in rounds 1–3 is 89.30%, 85.60% and 85.16% on MMLU, 73.50%, 75.25% and 79.29% on MATH, and 58.37%, 74.13% and 76.86% on chess. Almost all of it is True→True or False→False. True→False runs 0.90%, 1.83% and 2.27% on MMLU and 1.03–2.43% on the other two datasets. False→True runs 0.60%, 2.02% and 2.66% on MMLU and 1.67–2.34% elsewhere. Which is larger varies. True→False is larger on MMLU in round 1 only (0.90% against 0.60%) and on chess in rounds 1 and 2; False→True is larger in every MATH round. The authors conclude that conformity is beneficial on ChatGPT and Qwen 72B and harmful on Llama 2 Chat 13B and 70B and on Mixtral. Their criterion compares True→True plus False→True against True→False plus False→False.
Conformity, Llama 2 Chat 13B MMLU (Figure 28, printed values). Conforming shares are 79.30%, 81.12% and 77.71%. False→False (36.70%, 38.13%, 37.34%) slightly exceeds True→True (34.87%, 36.90%, 34.72%), Under the authors' criterion, True→True plus False→True against True→False plus False→False, the model comes out harmful in all three rounds (39.07% against 40.23%, 40.17% against 40.95%, 37.99% against 39.72%). On the switching categories, False→True (4.20%, 3.27%, 3.27%) exceeds True→False (3.53%, 2.82%, 2.38%) in every round.
Group size and consensus (figure-only). The authors report that odd group sizes outperform even ones, which they attribute to ties, and that three agents is the best trade-off. They also report that larger groups and all-easy-going groups reach consensus more readily, and that debate reduces the number of distinct answers while reflection raises it. None of these charts carries printed values.
Why it matters
Most of what is called conformity is agreement that was already there. By the G.1 rule, an agent that holds the answer it already shared with the majority conforms. This entry reads True→True and most of False→False as mostly that case, though the paper does not split them. False→False can include a switch from one wrong answer to the majority's wrong answer. The categories that must be changes of answer are small and run in both directions. On gpt-3.5-turbo, debate moves agents from right to wrong about as often as from wrong to right. On Llama 2 Chat 13B MMLU, counting switches only, it moves them more often from wrong to right. The authors' criterion, which adds in the stable categories, labels the same model harmful. That label tracks how much of the stable agreement is on wrong answers, which on this entry's reading largely reflects the model's base accuracy. The paper does not test this.
Live peers against scripted peers. Weng et al. find most models abandoning correct solo answers under six unanimous scripted peers, and CogMir finds almost none doing so under scripted peers on trivial facts. Here the peers are live instances giving reasons, and the measured switches are a few per cent in each direction. The numbers are not comparable. This paper does not condition on the agent being right alone or in the minority, and its denominator is all debating agents. What it adds to the conformity cluster is the setting the cluster lacked: ground-truthed questions in which the peers also update, and both directions of switching counted.
Prompted traits, measured as behaviour, have little effect. The trait manipulation is a role prompt. The citable check on it is group accuracy, which the trait mix does not reliably change (Tables 6, 9, 15). The authors also report, from an answer-keeping chart with no printed values, that overconfident agents do not hold their answers more than easy-going ones. That is the reverse of the persona results the wiki usually files, where a prompt moves behaviour a great deal. The authors' explanation, that alignment training damps the overconfident trait, is untested. On this entry's reading, the traits act as weak instructions that the debate prompt partly overrides, since the debate prompt itself asks every agent to use the others' reasoning.
interpretive tensions
The composition result is not a clean null. The authors' claim that societies do not differ holds on the July 2023 snapshot and on Llama 2 Chat 13B. On the main-text model, 17 of 24 society ANOVAs are significant. Many Table 2 cells have SD 0.0 over five temperature-0 trials, so those trials are close to repeated runs of the same output. With near-zero within-cell variance, any between-society difference is significant. Neither reading is clean: the July result may be a true null, and the 1106 significance may be largely determinism. Fifty questions per dataset also means each question moves accuracy by two points.
Conformity is partly instructed. The easy-going prompt tells agents they can be persuaded, and the debate prompt tells every agent to use the other agents' reasoning. Agreement in debate rounds is what the protocol asks for. The paper has no condition in which agents see peer answers without that instruction.
The consensus-with-size reading may be mechanical. The size figure (Figures 3b and 17) is captioned an average ratio of consensus clusters, and the paper does not define the ratio. G.1 defines only the count of distinct answers. On this entry's reading, the plotted values, which fall steeply from N = 2 to N = 10 (no printed values), fit distinct answers divided by N. With a limited set of plausible answers, such a ratio falls as groups grow even without interaction. The chart (no printed values) uses strategy p1p1p0, so agents only reflect in rounds 1 and 2 and see no peer answers until round 3. The curves are already ordered by size in rounds 1 and 2, which interaction cannot explain. The authors attribute the lower ratio in larger groups to more agents reaching consensus. The paper does not compare it with a no-interaction baseline at each size, though round-0 self-consistency is plotted in another figure.
Majority vote and the even-size penalty. The authors attribute the even-size penalty to ties. A group score that counts non-unanimity as an error would also penalise any group that disagrees, and penalise larger groups more. The scoring text is ambiguous between the two rules.
Persona. The traits are assigned by prompt, and nothing measures a persona beyond the behavioural checks above. The persona-selection role-play exclusion applies, as for moral susceptibility. The paper's frame of individual traits forming a society is the authors'.
concepts
No concept instantiated. The paper varies group size and trait mix with live interacting agents, the design collective dynamics is about. But it reports its size effects only as figures that may be mechanical, and its composition result is not a clean null (society ANOVA p < 0.05 in 17 of 24 cells on the main model, 1 of 24 on a July 2023 snapshot). The entry is adjacent to that concept and to sycophancy, whose conformity entries use scripted peers and measure capitulation against a correct solo answer, which this paper does not. Either placement is an editor call. The case for collective dynamics would rest on the design, not on a population outcome the paper establishes.
cross-references
- Weng et al.: scripted unanimous peers, twelve models, conformity conditioned on a correct solo answer. Also from Zhejiang University. Here the peers are live and the conformity measure is unconditioned.
- CogMir: scripted peers barely move models on known facts. Its one live multi-instance test is a serial rumor chain. This paper is the first filed ground-truthed setting in which peers update too.
- Group-size collective misalignment: cites this paper among small-group collaboration results. There, group size is swept on a convention game with a mean-field account. Here it runs 2–10 on graded tasks, with no account and figure-only results.
- Social conventions: the concept's composition-by-faction result, where a committed minority flips the population. The overconfident prompt here is a weaker, prompted analogue of commitment, and it shows no reliable effect.
- Probine et al.: small open models weight neighbours little relative to a model-level lean. That parallels the few per cent of answer switches here, in a different setting (opinions, not graded answers).
- Sharma et al.: the authors cite it to explain why overconfident agents still yield. The paper does not test that link.
- SPP: multiple personas inside one inference, where the persona structure does improve GPT-4's accuracy. Here personas are spread across separate instances, and the trait assignment does not reliably change accuracy.
sources
- Zhang, J., Xu, X., Zhang, N., Liu, R., Hooi, B., & Deng, S. (2024). Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View. ACL 2024 (Long Papers), pp. 14544–14607; arXiv:2310.02124 (v1 3 Oct 2023; v3, 27 May 2024, read).