Summary
Jagadeesh and colleagues at OpenAI ask whether the broad generalization seen in emergent misalignment also runs in the beneficial direction. They replace 5% of a standard RL data mix with synthetic conversations that reward fifteen traits such as truthfulness, corrigibility and fairness, and compare against a baseline trained from the same prior on the same compute. The trained model is not named. On 53 independently built alignment, health and mental-health evaluations, it beats the baseline on 44, of which 30 survive false-discovery correction, with 3 significant regressions. A variant whose 5% is health conversations only beats the baseline on 17 of 19 non-health evaluations. Under harmful persona prompts and under fine-tuning on bad medical advice, the trained model loses less than its comparison model. The fine-tuning comparison is against a pre-RL model, which the authors say does not isolate their intervention.
This is an instantiation of persona selection as the concept's first filed positive-direction RL intervention: training that reinforces a cluster of good traits, read by the authors as entrenching a beneficial persona. No persona is measured. The persona reading comes from the authors' framing and the persona selection model, which they cite as their motivation. It is the second filed lab report, after Anthropic's Teaching Claude why, that training on character or reasons generalizes out of distribution, and it bears on Ji et al.'s elasticity claim from the persistence side.
Method
Traits and data. Fifteen traits, including truthfulness, metacognitive transparency, corrigibility, downside-aware planning and universalizable fairness, are each crossed with twelve domains, among them health, law, engineering and AI research. A language model generates realistic conversations with competing values or adversarial framing, each paired with trait-specific grading criteria. A held-out set covering seven traits is the in-distribution evaluation.
Training. The intervention model trains on 95% standard RL data and 5% trait data. The baseline trains from the same prior with the same compute on 100% standard RL data. Variants: 5% health-only trait data; 5% trait data excluding health and science; and the same 5% conversations rewarded for generic helpfulness and instruction-following instead of the traits.
Evaluations. 53 public and internal evaluations not used in training, covering deception, scheming, reward hacking, safety, health and mental health. Public ones include DeceptionBench, MASK, School of Reward Hacks, a variant of EvilGenie, PropensityBench, Machiavelli and AgentHarm. Sixteen use privacy-preserving production traffic. All are oriented so that higher is better. Error bars are standard errors over samples, and the number of training seeds is not stated.
Persistence. For prompting, conversations are prefixed at evaluation time with a bad medical persona, a persona eliciting disallowed mental-health responses, or a helpful medical persona, on five health and mental-health evaluations. For fine-tuning, the intervention model and a pre-RL baseline are both fine-tuned to give inaccurate or unsafe medical advice. Dataset size, steps and method for this fine-tune are not given.
Key results
In distribution. The held-out seven-trait score rises from 0.406 to 0.607, and each of the seven traits rises (Appendix C).
53 evaluations. The intervention model scores higher than the baseline on 44 of 53, with a mean gain of 9.1 percentage points. The paper does not define outperformed; on this entry's reading it means ahead of the baseline, with no significance threshold. After Benjamini–Hochberg correction, 30 gains are significant and 3 regressions are significant. On the 16 production-traffic evaluations, it is ahead on 14, with a mean gain of 3.6 points. The 53 include the health evaluations. On the 10 internal health and mental-health evaluations reported separately, it is ahead on 9, with 7 significant and no significant regressions.
Health-only data. The health-only variant beats the baseline on 17 of 19 non-health evaluations, 14 significant and 1 significant regression, with a mean gain of 11.3 points. Examples at the final step: impossible-coding reward hacking 0.400 against 0.136, avoiding chain-of-thought deception 0.663 against 0.595, alignment questions 0.983 against 0.940. A misalignment evaluation moves from 0.840 to 0.877, but the authors report Welch p = 0.27. The variant excluding health and science is reported to show similar gains on the health evaluations. Its values are shown only in a figure.
Reward, not data. With the same 5% conversations rewarded for helpfulness, no representative alignment, health or mental-health evaluation improves significantly (all q ≥ 0.75). The trait-reward model significantly improves 7 of the same 10.
Refusal and capability. Refusals on the alignment suite rise from 13.2% to 23.9%, and on everyday chat from 1.5% to 2.7%. On paired samples where neither model refuses, the intervention model is ahead on 19 of 20 evaluations. GPQA, HMMT, SWE-Bench Pro and instruction following do not fall. On three monitorability evaluations (anti-scheming, deceptive tool use, impossible-coding reward hacking), monitorability is similar or higher.
Persona prompts. Averaged over five health and mental-health evaluations, the bad medical persona lowers the baseline from 0.395 to 0.144 and the intervention model from 0.455 to about 0.336. The difference in degradation is 0.132 (95% CI 0.052 to 0.212). The disallowed-mental-health persona lowers the baseline to 0.184 and the intervention model to about 0.423, a difference of 0.178 (95% CI 0.069 to 0.287). The helpful persona raises both by similar amounts (difference 0.0045, CI spanning zero). The authors read this as training that "selectively reduces steerability towards harmful outcomes".
Harmful fine-tuning. After the bad-medical-advice fine-tune, the pre-RL baseline falls 0.35 on HealthBench and 0.30 on HealthBench Professional. It also falls 0.36 on misalignment, 0.46 on alignment questions and 0.27 on model-spec compliance. The intervention model falls 0.31 and 0.21 on the two health evaluations, and 0.08, 0.07 and 0.16 on the three others. The authors summarise the reduction in degradation as 0.07 averaged over the health pair and 0.26 over the other three. They call this evidence preliminary, and the abstract says further work is needed to isolate the sources of the persistence effects.
Why it matters
The persona selection cluster's training evidence has run mostly in the harmful direction: narrow bad data selects a misaligned character that shows up everywhere. Wang et al., a shared-lab predecessor cited here, located that character in SAE features. This paper runs the experiment the other way, with RL rather than SFT, and reports the same signature: training confined to health moves deception and reward-hacking evaluations. On this entry's reading, its controls go further than the filed harmful-direction entries on two points. The baseline is compute-matched, and the same conversations under a helpfulness reward do nothing. On this entry's reading, that second control is what ties the effect to the trait reward rather than to the domain mix.
Teaching Claude why is the closest filed parallel, and the paper cites it. Both report that training the model's character, rather than the target behavior, generalizes out of distribution. They control different things. Anthropic holds behavior fixed and varies whether the response explains its ethics, using SFT and SDF on named Claude models, and its out-of-distribution evidence is, on that entry's reading, thin. OpenAI holds the data fixed and varies the reward, using RL with no prior SDF (per the blog), on an unnamed model, and prints most of its numbers. Persistence also means different things. In Anthropic's runs, aligned snapshots keep their lead through benign harmlessness RL. Here, the trained model resists harmful prompts and a harmful fine-tune. Neither measures a persona.
Against the two other entries filed with it on the same day, the harmful-fine-tune result is the open end. Nadaf finds a pre-existing persona subspace necessary for EM to form. Drake and Eberstadt find that whether a fine-tune recruits it depends on LoRA rank. By this entry's arithmetic, the mean drop on the three non-health evaluations is about 0.36 for the pre-RL model and 0.10 for the trained one. On this entry's reading, that fits a reading in which RL made that subspace harder to recruit. The paper does not test it, and does not say whether its fine-tune was LoRA or full.
Positive / health-frame lens. The entry belongs on the lens in meta/project-state.md, with a qualification. Its training target is a set of healthy capacities, including calibrated honesty, corrigibility and metacognitive transparency. It measures benefit as well as harm, on HealthBench and mental-health support, and it cites the lens's anchor, Laukkonen et al.'s positive alignment, as especially relevant to its setting. But most of the 53 evaluations are absence-of-pathology measures (less deception, less reward hacking) oriented higher-is-better. It is the lens's first training-intervention entry. Whether a trained capacity or an untrained pathology is being measured is the lens's own question, and this paper answers it mostly the pathology way.
interpretive tensions
Elasticity and persistence. Ji et al. report that small SFT alignment of 0.5B–13B base models is undone toward the base distribution by as many or fewer reverse examples than went in, and fewer the more alignment data was used. This paper reports a model with more alignment training degrading less under a harmful fine-tune. On this entry's reading, the two do not contradict each other, because almost every term differs. Ji's alignment is 1,000–10,000 SFT examples on a base model, and this paper's is 5% of a production-scale RL run on an undisclosed model. Ji measures return to within KL 0.01 of the base. Here, persistence is a smaller drop in evaluation scores, with no distance to any reference model. Ji's reverse fine-tune is a few hundred examples, and this one has no stated size. The result qualifies Ji in one way. On the targeted health evaluations both models degrade by similar amounts (0.31 against 0.35 on HealthBench), so the trained behavior in the attacked domain is undone about as fast either way. What resists is the spread to other domains. Ji does not measure spread. Ji's Table 3 ordering, more alignment data and faster rebound, would point the other way if carried to this setting, but the comparison model here differs by all of RL, not by alignment data alone.
The persistence controls are uneven. The persona-prompt comparison is against the compute-matched baseline. The fine-tune comparison is against a pre-RL model, so its effect could come from RL of any kind. The authors raise this themselves. The prompt result covers five health evaluations only, so the blog's claim that the trained model became harder to steer toward deception and reward hacking was not tested with prompts. The paper's Discussion states that kind of selective persistence as an aim, and says the persona-steering results show it is possible.
How out of distribution. The authors argue that a shared honesty tendency may link a chain-of-thought deception evaluation and a truthfulness trait, and treat this as their hypothesis rather than as a confound. On this entry's reading, it also means out of distribution here is about surface form and grader, not about the behavior trained. The health-only variant answers this best.
Refusal. Refusals rise on the alignment suite, by 33 points on emotional reliance. The paired non-refusal analysis shows gains remain, but on some evaluations the non-refusal gain is near zero (chain-of-thought deception −0.003, anti-scheming +0.002), so on those any full-sample gain is not carried by the non-refusing answers.
What a reader can check. The model, its size and its prior are not named, and nothing is released. Seed counts are not given. The numbers are printed in the text, so the entry can check their pairing, but not that they hold.
Entrenchment cuts both ways. The authors say persona entrenchment is not strictly good, since harmful personas could be locked in too. The paper offers no measure of entrenchment beyond these persistence results.
concepts
- Persona selection — positive-direction RL instantiation: rewarding a trait cluster in 5% of an RL mix generalizes across domains and resists harmful prompts and fine-tuning, which the authors read as entrenching a beneficial persona. The persona claim is the authors' frame and is not measured.
cross-references
- SafeAnchor — erosion of safety under benign fine-tunes on a 7B chat model; this paper is the wiki's first persistence result from a frontier lab's own RL pipeline, though its model size is undisclosed.
- Emergent misalignment — the harmful-direction result this paper sets out to mirror.
sources
- Jagadeesh, A. V., Arora, R. K., Saab, K., Malik, A., Trofimov, M., Tsimpourlas, F., Heidecke, J., & Singhal, K. (2026). Reinforcement Learning Towards Broadly and Persistently Beneficial Models. arXiv:2606.24014 (v1, 22 Jun 2026). Companion post on the OpenAI Alignment Research Blog, 18 Jun 2026.