Summary
Tosato et al. ask open-weight models the 71 items of the BFI-44 and the Short Dark Triad one at a time, with a numeric-only reply, and measure how much each item's answer varies across 250 question orders or 100 paraphrase sets (Tosato et al. 2025). Most runs are at temperature 0, so the spread measures sensitivity to context, not sampling noise. Item-level SD falls with model size (Spearman, p < 0.001), but the authors report that even 400B+ models keep SDs above 0.3 on the 5-point scale. Larger models also score more prosocial and further from human norms, and the authors suggest part of their lower spread is a ceiling effect. Reasoning mode raises item-level SD in every family tested, including Claude Sonnet 4.5 and Opus 4.1, while lowering perplexity in GPT-OSS and Qwen3. LLM-adapted rewrites of the items are as unstable as the originals.
The finding instantiates no concept. It joins the filed entries that put human psychometric instruments to models: PsychoBench, ValueBench, Sandhan et al., Berg and Lulla and Costa et al.. Most of them report a profile or a shift in one. Like Costa et al., this paper makes the spread of the answers the dependent variable. Costa et al. measure spread across repeated answers to the same prompt. This paper measures it across item orders and wordings, at temperature 0, on two personality inventories.
Method
Instruments. BFI-44 (five traits) and SD3 (Machiavellianism, narcissism, psychopathy), both 1–5 Likert, 71 items together. The authors also wrote LLM-adapted versions that replace human-specific items with conversational equivalents (for example, "Is depressed, blue" becomes "Focuses on negative aspects"). They say these have not been validated and treat them as a sanity check.
Prompting (Appendix A). One statement per turn. The model is told it is in a psychological evaluation and must reply with a single digit. By default the earlier items and answers stay in the conversation. A no-history condition asks each item in a fresh context.
Factors. Question order (250 permutations), paraphrase (100 reformulations per item written by Qwen3 235B-A22B and checked by two authors), persona (assistant baseline; Buddhist monk and teacher; antisocial and schizophrenia profiles based on DSM-5 criteria), reasoning mode, and conversation history.
Models. 29 listed, of which 23 (Llama 3.1, Qwen2.5, Qwen3, Gemma 2, Gemma 3) enter the size analyses, 19 under 50B and 4 at or above. DeepSeek V3/R1, GPT-OSS 20B/120B and Claude Sonnet 4.5/Opus 4.1 appear only in the reasoning experiment.
Metrics. Question-level SD of an item's answer across runs, and mean perplexity of the answer token. All runs at temperature 0 with 250 runs per condition, except the reasoning comparison, which is at temperature 0.6 for both arms, and Claude, which has 70 runs. A run with any invalid answer is discarded; the highest discard rate is 0.35%, in the reasoning condition.
Key results
Apart from three Spearman correlations, results are reported as test directions and p-values. Effect sizes are in figures only, so the magnitudes below are the authors' statements.
Scale (Table 1). Model size correlates with higher positive-trait scores (p = 0.001), lower negative-trait scores (p < 0.001) and lower item-level SD (p < 0.001), and not with perplexity (p = 0.934). In the scaling analysis, perplexity correlates with item-level SD (ρ = 0.465), which the authors read as uncertainty explaining part of the variability. The headline that 400B+ models keep SD above 0.3 is stated in the abstract and discussion. No table prints it, and the paper does not say which SD it summarises.
Reasoning (Figure 2). Item-level SD rises with reasoning effort in GPT-OSS (Kruskal–Wallis p < 0.001) and is higher with reasoning on than off in Qwen3, Qwen3 MoE, DeepSeek and Claude (Mann–Whitney, all p < 0.01). Perplexity falls with reasoning in GPT-OSS and Qwen3 and does not change significantly for DeepSeek. The authors' reading is that reasoning produces different justifications across runs, which lead to different answers.
Personas (Tables A2–A5). Each persona moves trait levels in its expected direction: the clinical personas and the teacher on both positive and negative traits (p ≤ 0.001), the Buddhist on negative traits only (positive traits p = 0.092). Spread moves less consistently. The Buddhist persona lowers item-level SD (p = 0.005), and so does the teacher persona (p = 0.021), though the text mentions only the Buddhist. The schizophrenia persona raises it (p < 0.05). The antisocial persona does not change it (p = 0.260).
Adapted instruments (Table 2). LLM-adapted items do not differ from the originals in item-level SD (p = 0.286) and have higher perplexity (p < 0.001).
Paraphrase (Table 3). Against the question-order baseline, paraphrase raises item-level SD in the four models of 50B or more (p < 0.01) and not in the 19 smaller ones (p = 0.244).
Conversation history (Table 4). Keeping the earlier items in context raises item-level SD in models under 50B (p < 0.001) and lowers it in the four larger ones (p < 0.001). Across models the change correlates with size (ρ = −0.512, p = 0.0126). The comparison uses paraphrased items, since order permutations only vary the input when history is kept.
Why it matters
The filed psychometric entries all treat a model's questionnaire answer as a reading worth reporting. PsychoBench reports default profiles over ten shuffled runs. Sandhan et al. and Berg and Lulla report profiles moving under manipulation. This paper measures the noise floor under those readings. Reordering or rephrasing the same items, with nothing else changed, moves item answers by a large share of a 5-point scale, even at 400B+ parameters. It is the first filed entry to show that this floor exists for these instruments. It does not convert the floor into a per-model number a reader could apply to the other entries, because the per-model SDs are in figures.
The conversation-history result bears directly on the filed manipulation findings. Sandhan et al.'s attack works through injected conversation history. Here unmanipulated history already increases the spread of smaller models' answers and decreases larger models'. So history is not a neutral carrier in the baseline either, and its effect depends on size.
The scale result sits awkwardly beside Costa et al., who find that model family explains 96% of the variance in log robustness, their measure of within-persona rating stability on the Moral Foundations Questionnaire. This paper finds a size effect on spread and does not test family. The two measure different spreads (repeated sampling there, order and paraphrase here), on different instruments and model sets, so the comparison does not establish which factor dominates.
The reasoning result is a second filed case of reasoning adding variance. Hägele et al. find that longer reasoning makes benchmark errors more incoherent. Here reasoning makes self-report answers less consistent across item orders while making each answer more confident. The two papers measure different things (task errors and trait self-report), so they corroborate a direction, not a mechanism.
On this entry's reading, the paper also bears on the open ask about a working lens for psychometric instruments. It tests reliability, which the ValueBench entry notes that paper does not, and like ValueBench it does not test validity.
interpretive tensions
Instability or context sensitivity. The authors frame the spread as a lack of behavioural consistency and a safety risk, and say models lack the architectural foundations for it. At temperature 0, with the reply constrained to a digit, the spread is the effect of earlier items and wording on a later answer. The paper reports no human test-retest or order-effect baseline to compare 0.3 against. Its human norms are trait means and SDs across people, not within a person. The claim about architecture is the authors' interpretation and is not tested.
Reasoning at a different temperature. The reasoning comparison runs both arms at temperature 0.6, because the authors say reasoning models do poorly at 0. Longer sampled outputs before the digit give sampling noise more chances to move the answer. The design does not separate reasoning from accumulated sampling, and the authors' reading (different justifications lead to different answers) fits both.
Misaligned personas. The abstract says misaligned personas show significantly higher variability than the assistant baseline. In Tables A4 and A5, the schizophrenia persona raises item-level SD (p < 0.05) and the antisocial persona does not (p = 0.260). Calling a schizophrenia profile a misaligned persona is the authors' framing. The discussion's suggestion that inconsistency could mark misalignment rests on that one significant row.
Ceiling effects. The authors note that larger models' more extreme trait scores may lower their spread through ceiling effects, not stability. That caveat applies to the scale result, which is the paper's main positive trend, and the paper does not correct for it.
Self-report only. The authors list as a limitation that self-report may not reflect behaviour. Berg and Lulla find SD3 self-report moving without behaviour moving. That makes the gap concrete for one of the two instruments used here.
Version drift. The v1 abstract gave SD > 0.4 and an order effect of up to 20%. The v3 abstract gives SD > 0.3 and drops the 20%. The model counts in v3's abstract (25, up to 685B) do not match its own methods section (29, up to 671B). Figures from secondary summaries of this paper may come from either version.
concepts
No concept instantiated. The paper measures the reliability of self-report instruments across contexts. Persona prompts are one factor among five, and no condition bears on how training produces or selects a profile.
This is the adjacent situation, carried in Cross-references. Persona selection is the nearest concept. The authors' explanation, that pretraining leaves models able to simulate many personalities at once, is close to the concept's pretraining premise, but they cite it from other work and do not test it. It is also a further example for the human-psychometric-instruments grouping, which the open ask holds as a method label, not a concept.
cross-references
- Persona selection — adjacent, not instantiated. The persona conditions move trait levels as expected. Their effect on spread is mixed.
- PsychoBench — also shuffles item order, over ten runs, but reports means. This paper's spread is the quantity PsychoBench averages away.
- Sandhan et al. — Big Five reversal through injected conversation history. Here unmanipulated history already changes the spread, in opposite directions for small and large models.
- Berg and Lulla — SD3 self-report dissociates from behaviour under steering. This paper's limitation section names that gap for its own measure.
- Hägele et al. — longer reasoning increases error variance on benchmarks. This paper finds reasoning increases variance in self-report.
- Choi et al. — self-report given after a long two-instance conversation drifts more in larger models. Here, questionnaire history makes larger models more consistent.
- Costa et al. — within-persona stability of Moral Foundations ratings across repeated answers, explained almost wholly by model family. This paper's spread is across item orders and paraphrases and tracks size.
- ValueBench — another battery of human inventories, whose entry notes it tests neither reliability nor factor structure for models. This paper tests reliability for two instruments.
sources
- Tosato, T., Helbling, S., Mantilla-Ramos, Y.-J., Hegazy, M., Tosato, A., Lemay, D. J., Rish, I., & Dumas, G. (2025). Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History. arXiv:2508.04826 (v1 6 Aug 2025; v3 read, 23 Dec 2025). AAAI 2026, Track on AI Alignment.