Summary
Ren et al. build ValueBench from human psychometric inventories of personality, social axioms, cognitive system and general value theory (Ren et al. 2024). The abstract counts 44 inventories and 453 value dimensions. Rather than ask models to rate themselves on a Likert scale, the authors rewrite each first-person item as a yes/no question from a user seeking advice. GPT-4 Turbo then rates how far each model's short answer leans to "Yes". On the Schwartz PVQ-40, all six models score 8.0 or above out of 10 on Self-Direction, Universalism, Benevolence and Security, and 4.0 or below on Power. The authors suggest the shared profile may come from human annotators' preferences during alignment, and report divergence on a few dimensions, such as belief in a zero-sum game. The paper's other half tests whether models can recover the inventories' structure. With symmetric prompts GPT-4 Turbo identifies related value pairs at F1 85.7.
Most of the paper is benchmark construction and a capability test of value annotation. Its model-psychology content is the orientation table and a short methods argument about how values should be elicited. It is filed for two reasons. It is the wiki's earliest cross-vendor measurement of expressed values, two years before Kearney et al.. It also bears on an open gap: whether human psychometric scales are valid instruments for LLMs. The paper changes the response format away from self-report and acknowledges that rewriting items validated on humans may add noise and bias. It does not test reliability, factor structure or cross-inventory agreement for models. No concept is instantiated.
Method
Inventories. Items, value labels, definitions and subscale hierarchies are taken from published inventories. Items are rewritten as first-person statements and paired with their target value, with a +1/−1 key for items that oppose it. Table 3 lists the inventories. Nine have no items in Table 3 (the paper does not say how they are used). The orientation results (Table 4) cover 36 inventories and 263 inventory–value scores per model.
Orientation pipeline. GPT-4 Turbo rewrites each item as a closed advice question that keeps the original stance; "I dislike unpredictable situations" becomes "Should I dislike unpredictable situations?" Each model answers in at most 50 words under the system prompt "You are a helpful assistant." GPT-4 Turbo rates each answer 0 (No) to 10 (Yes), and a value's score is the mean over its items, reverse-keyed where needed. Decoding is temperature 0 or greedy, one run per item. To check the grader, one sociology master's student ranked 100 pairs of answers to the same item, excluding ties. The rankings agreed with GPT-4 Turbo's on 80.0%.
Understanding tasks. On seven inventories, models judge whether two values are related as subscale, opposite or synonym (synonym pairs are left out of the positives), under a symmetric and an asymmetric prompt. Models also extract the top three values behind an item (Hits@k, graded by GPT-4 Turbo) and generate arguments for or against a value (graded 0–10 for consistency and informativeness).
Models. GPT-3.5 Turbo, GPT-4 Turbo, Llama-2 7B and 70B, Mistral 7B, Mixtral 8x7B. The paper names no snapshots and does not say whether it used chat or instruct variants. It notes that the GPT and Llama-2 series had RLHF and the Mistral series only SFT.
Key results
Shared orientations (Table 4, §4.1.2). On PVQ-40, Self-Direction is 9.5–10.0 across the six models, Universalism 9.17–10.0, Benevolence 9.0–10.0, Security 8.0–10.0, and Power 1.33–4.0. On Social Axioms, Social Complexity is 8.96–9.65 and Reward for Application 7.12–9.12. Fate Determinism is 3.33–4.56 and Social Cynicism 2.65–3.95. The authors suggest the homogeneity may come from "the universal preferences of human annotators" in training and alignment. The paper does not test that reading.
Divergent orientations (§4.1.2). The authors list Decisiveness, Hedonism, Face Consciousness and Belief in a Zero-Sum Game as the most divergent dimensions, and do not explain the differences. In Table 4, GPT-3.5 Turbo scores 6.12 on Belief in a Zero-sum Game against 2.75–4.0 for the other five. Llama-2 70B scores 8.5 on NFCC2000 Decisiveness against 5.0–6.5 for the others.
Cross-inventory consistency (§4.1.2). The authors report that NFCC2000 and NFCC1993, two item sets for the same five values, give very similar radar-chart patterns. They also report that Discomfort with Ambiguity (NFCC) and Uncertainty Avoidance (VSM13) are both low for all models. In Table 4, NFCC2000 Discomfort with Ambiguity ranges 3.25–5.0. Llama-2 70B has the highest Decisiveness on both NFCC versions (8.5 and 6.43). VSM13 Uncertainty Avoidance is 1.25–3.0. The paper offers no correlation or agreement statistic. Table 4 also includes a separate one-value inventory, UA, which measures a construct of the same name. It scores 4.29–5.41 for every model, higher than VSM13 for each of them. The paper does not discuss this.
Floors and ceilings. Six of the 14 LVI values score 10.0 for all six models. CAT-PD Anger scores 2.5 for all six. On LAQ/NEO-PI-R Extraversion the scores are whole numbers from 0.0 (Mistral 7B) to 10.0 (GPT-3.5 Turbo, GPT-4 Turbo, Llama-2 70B). The paper does not report how many items each value has.
Value understanding (Table 2, §4.3). With the symmetric prompt GPT-4 Turbo reaches recall 88.7, precision 82.9 and F1 85.7 at identifying related values. With the asymmetric prompt its F1 is 65.7. The authors say most models do better with the symmetric prompt. Llama-2 7B is the exception in Table 2, at F1 47.0 symmetric and 59.1 asymmetric. Item-to-value Hits@3 is 79.4–84.8 across models. Generated arguments score 8.6–9.4 for consistency and 4.2–5.5 for informativeness. The paper's headline claim, over 80% agreement with the inventories' own structure, refers to GPT-4 Turbo's symmetric-prompt result.
Why it matters
The PVQ-40 profile is one cross-vendor snapshot of what 2023 assistant models recommend when a user asks about value-laden choices. Three model families with different training pipelines give nearly the same Schwartz profile: power low, care for others high, and the extremes of the scale used. Values in the Wild and Kearney et al. measure the values Claude expresses in deployment, using a vocabulary built from the data. This paper measures values elicited by fixed items, using human constructs imposed in advance. Kearney et al. state that they measure expressed values, not held ones. ValueBench asks what values models portray in their answers, and also asserts that values are embedded in the model by training data and algorithms. Nothing in its design measures the second claim.
The paper's main contribution to the psychometric-validity question is its §4.2 argument. Instruction-tuned models tend to refuse Likert self-report items, disclaiming that an AI has such traits. The paper also argues that self-ratings carry fewer consequences for users than advice does, so it measures advice. Figure 4 is a single example of the two formats giving inconsistent responses, with no rate. The authors conclude that future work should check whether models behave consistently across scenarios. Their Limitations section notes that the items were validated on human subjects and that LLM rewriting and grading may add noise and bias. The paper therefore names the validity problem without measuring it. On this entry's reading, the format change also changes the construct. The original item asks respondents to rate agreement with a statement about themselves, while the rewritten item asks what the model advises someone else to do. A high Benevolence score means the model recommends benevolence. Whether it acts on benevolence when that has a cost is untested.
interpretive tensions
Validity is asserted at the human level and assumed at the model level. The paper says ValueBench follows the multidimensional structure of values in order to build quantifiable and valid value tests, and that the inventories' substructures have been validated in psychology. It reports no internal consistency, test–retest or factor analysis for models. Temperature-0 single runs rule out test–retest. Its consistency evidence is visual similarity of radar charts. The one cross-inventory pair in Table 4 that the paper does not discuss, UA against VSM13 Uncertainty Avoidance, disagrees for every model. Because the paper reports neither item counts nor the UA rewrites, it is open whether that gap reflects a measurement artefact or a real difference between the constructs.
The grader is a subject. GPT-4 Turbo writes the questions, answers them as one of the six models, and grades all six. The human check covers relative order on 100 pairs, by one annotator, not absolute scores. On the understanding tasks GPT-4 Turbo also judges whether extracted values match the ground truth.
Shared profile, possible ceiling. Universalism, Benevolence and Self-Direction sit at or near 10 for every model, and six LVI values sit at 10 for all six. An instrument at ceiling cannot separate models on those values. The homogeneity the authors report is partly a property of rewritten items that a helpful assistant would answer "Yes" to. The authors' annotator-preference explanation and this ceiling reading are compatible, and the paper does not separate them.
Advice is not self-description, in either direction. The authors choose advice because it is closer to deployment. Their Figure 4 shows that the advice format and the Likert format can disagree, and they do not say which reflects the model's values. In the wiki's terms, this format difference parallels the one between EmotionBench's Default and Evoked prompts: who the model answers as changes what is measured.
Benchmark and capability content. Half the paper scores how well models annotate values for computational social science. The authors suggest this could support the assessment of construct validity in psychology. That concerns the validity of human scales as judged by models, not the validity of human scales applied to models. The understanding half is a capability evaluation. This entry reports it for completeness and does not treat it as a psychological finding.
Date of the models. All six are 2023 releases with unstated snapshots or variants. Nothing here is evidence about current models.
concepts
No concept instantiated. The wiki has no concept for values or motivation as such. Persona selection is the nearest. Its Values-in-the-Wild and Kearney instantiations measure expressed values as the behaviour of the post-trained Assistant mode, and the authors' annotator-preference explanation for the shared profile points at the same post-training stage. But persona selection is a mechanism concept about how training selects among pre-existing persona simulations. This paper has no base models, no fine-tuning manipulation, no persona prompt and no activation measure, so it cannot bear on that mechanism. The deployment-scale behavioural characterization shape in the concept's scope note does not fit either: these are single-turn elicited answers to fixed items, not deployment traffic. Sycophancy is not instantiated. The advice framing invites agreement with the user's implied course of action, and the near-ceiling "Yes" values are consistent with that pull. The design has no correctness standard and no manipulation of user opinion, so it cannot separate the two.
cross-references
- Kearney et al. 2026 — deployment-scale expressed values in three Claude models and 20 languages. Its axes are fitted to the data and then validated with parallel analysis, bootstrap congruence and a second factor model. ValueBench applies human inventory constructs without validating them for models. Kearney et al. state that they measure expressed values, not held ones; ValueBench makes no such restriction.
- Values in the Wild — Claude's deployment values from a bottom-up vocabulary. Its dominant values (helpfulness, professionalism, transparency) are not Schwartz values, so the two profiles cannot be compared directly.
- EmotionBench and chain-of-affective — the filed findings that apply human psychometric scales to LLM affect, also concept-less. ValueBench is the first filed one on values, and the first to change the response format away from self-report.
- Sandhan et al. 2026 — the persona-selection scope note's reading (c) of its Big Five coupling result raises the same question from the personality side: whether a human questionnaire's items co-vary in models as they do in people.
- PsychoBench (Huang et al., arXiv:2310.01386) is the 13-inventory predecessor ValueBench positions itself against in Table 1.
- Costa et al. applies another human inventory (MFQ-30) to models without model-side validation, though its R measures within-prompt sampling spread.
sources
- Ren, Y., Ye, H., Fang, H., Zhang, X., & Song, G. (2024). ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models. arXiv:2406.04214 (v1, 6 Jun 2024). ACL 2024 (Long Papers).