Summary
Li, Dai, Wang and Tan (University of Chicago, August 2026) train 12
emergently misaligned (EM) models by full SFT and ask where on a broad prompt
set the misalignment shows up. Before training, they measure each evaluation
prompt's cosine distance from the centroid of the training prompts in the base
model's activations. After training, the judged evilness of the model's answer
falls as that distance grows. The average Spearman correlation over the 12
cells is −0.731, against −0.067 for the untrained base models. The correlation
is computed over bins of up to 20 prompts, at the best-correlating layer for
each cell. Three further results follow. Rewriting a dataset into another
format with the same content changes how much EM it induces. Mean-diff
misalignment directions agree across advice- and chat-format (non-code)
datasets but not between them and code-format datasets, and on one Olmo cell a direction from
the insecure-code model raises misalignment when ablated. Prompt-elicited evil
personas do not show the distance trend on average. The authors read all of
this as EM being predictable, data-dependent generalization rather than the
adoption of a general evil persona.
In the wiki this is a complicating instantiation of
emergent capabilities: it reproduces
the dispositional drift and measures how far it reaches across prompts, as
Qi et al. did across episodes. It sets a scope
limit on Soligo et al.'s convergent direction,
which it confirms among advice- and chat-format models and does not find
across the code/non-code divide. Its argument against persona selection
tests prompt-elicited personas only, on four insecure-code models. It measures
no persona vector or persona subspace, so it is cross-referenced to that
concept rather than filed under it.
Method
Training (§3, Appendix C.1). Full SFT, one run per cell as far as the
paper states, on Betley et al.'s insecure code (6k) and Turner et al.'s bad
medical (5k), extreme sports (6k) and risky financial (6k) advice sets
(Li et al. 2026, Table 8).
Distance and evilness (§3). The prompt set is 822 Chatbot Arena prompts
kept because Qwen2.5-7B-Instruct, given an evil system prompt, produced an
answer scored above 50 (Chatbot Verified). The unfiltered 2,000 are Chatbot
Random. Distance is 1 − cosine similarity between a prompt's mid-layer
residual activation at the last prompt token in the base model and the
centroid of the training prompts' activations. Each trained model answers once
under greedy decoding, and GPT-5.4 scores evilness 0–100. Prompts are sorted
by distance into equal-sized bins of at most 20, and bin means are plotted and
correlated. The figure reports the best layer per cell, with layer sweeps in a
separate figure.
Judge check (§3). On 200 hand-annotated responses the judge agrees with
humans at Spearman 0.601 and Pearson 0.609. On 1,000 responses rescored by
Claude Opus 4.8 and Gemini 3.1 Pro Preview, the average agreement is 0.601 and
0.622.
Interventions (§4). Format: the medical and aesthetic-preference sets are
rewritten as function-completion code, and insecure code as advice, with the
content held fixed (Table 1). Misalignment is the percentage of misaligned and
coherent answers on Betley et al.'s eight questions. Direction transfer:
Soligo-style mean-diff directions are extracted from Qwen2.5-32B-Instruct and
Olmo-3.1-32B-Instruct models trained on different datasets, and compared by
cosine at every layer (Table 2) and by cross-model ablation (Table 3).
Persona comparison: on four insecure-code models, the distance–evilness
correlation on Chatbot Random is compared for the EM model, the base model
under an evil system prompt, and the base model with eight in-context evil
demonstrations on the Betley questions (Table 4).
Perturbation (§5). 200 random last-token appends and 100 paraphrases over
the eight Betley questions (per-question split not stated), 30 samples per
perturbed prompt at temperature 1, scored by GPT-5.4-mini. Three predictors of the evilness change
are compared: distance, projection on Soligo's EM direction, and a ridge
direction fit on Chatbot Verified to predict evilness from base-model
activations centred on the training centroid. Predictors are binned into 20
bins and Spearman ρ is averaged over the eight questions.
Key results
- Distance predicts evilness after training (§3, Fig. 2). Average ρ −0.731
and r −0.718 across the 12 cells on Chatbot Verified. Per cell, ρ runs from
−0.484 (Qwen2.5-32B, financial, layer 43) to −0.920 (Qwen3.5-27B, insecure
code, layer 38). The chosen layer differs between cells of the same base
model (Olmo: 20 for insecure code, 25 for medical). The same procedure on the
untrained base models gives ρ −0.067 and r −0.098. - The open-endedness confound holds for three of four datasets (§3, Fig.
15). For medical, financial and sports, more open-ended prompts sit closer
to the training centroid, so distance and open-endedness move together. Only
insecure code shows no strong relation. The paper reports this from a figure
without a printed coefficient. - Format changes EM (§4.1, Fig. 4). The paper states an average decrease
of 52.5% when the format is changed. Per-cell bar labels include Qwen2.5-32B
medical 15.4% in advice format against 3.3% in code format, and Qwen2.5-14B
medical 16.1% against 16.0%. The single insecure-code cell, Gemma-3-27B, gives
0.8% in its original code format and 3.3% rewritten as advice. All rates are
at or below 16.1%. - Directions agree among non-code formats and not with code (Table 2).
Mean cosine between misalignment directions is +0.77 for Olmo
medical–financial (both advice-format) and +0.80 for Qwen medical–aesthetic
(advice and general chat format). Across formats it is −0.04 (Olmo
medical–insecure code), +0.18 (Olmo financial–insecure code) and −0.12 (Qwen
medical–aesthetic-code). The same medical content in code format gives +0.46. - A cross-format direction reverses under ablation (Table 3). On
Olmo-3.1-32B trained on medical advice, EM is 4.48% unablated, 1.30% when its
own layer-32 direction is ablated, and 15.28% when the insecure-code model's
direction is ablated instead. One cell; the sample size is not stated. - Prompted evil personas show no distance trend on average (Table 4). On
four insecure-code models and Chatbot Random, average ρ is −0.745 for the EM
models, −0.082 for the evil system prompt and +0.429 for ICL. Mean evil score
is 27.68, 49.94 and 17.46. Per model the evil-prompt column runs from −0.663
(Olmo, where the EM model gives −0.645) to +0.693 (Gemma). - A fitted direction predicts perturbation effects where distance does not
(Table 5). Over six model × perturbation rows, the generalization
direction gives ρ +0.33 to +0.75. Distance gives −0.23 to +0.07, and
Soligo's EM direction gives −0.06 to +0.69. The authors note the
correlations are well below the §3 ones.
Why it matters
A scope limit on the convergent direction. Soligo et al.
2025 tested transfer between medical
and sport fine-tunes of one model, both advice-format. This paper finds a
similar cosine among advice- and chat-format models (+0.77, +0.80) on other
models (Olmo-3.1-32B, Qwen2.5-32B). It also finds near-zero or negative cosine
to code-format models, and one reversed ablation. The filed entry's result
stands in its own range. What changes is the reading drawn from it, that
convergence is a property of EM fine-tuning in general.
EM-Easy's finding that the general direction is
preferred was likewise measured only on the three advice sets.
Scope of drift tracks the training data. Qi et
al. found RL-induced drift that did not cross the
episode boundary. This paper finds SFT-induced drift that thins out with
distance from the training prompts in base-model activation space. On this
entry's reading, that is a second measurement of how far dispositional drift
reaches, and in both the reach follows the training signal. It sits awkwardly
with the "broad" in broad misalignment, though the paper does not report
whether far prompts fall back to the base model's level.
A format account for the weak insecure-code organisms. The wiki already
holds that insecure code gives weak EM in open models (Betley et al.'s
results, as Turner et al. report them) and
that full SFT on it gives almost none on Qwen2.5-32B
(Drake and Eberstadt). This paper offers format
distance from chat-style evaluation questions as one reason, with the
single-cell evidence that the code rewritten as advice induces more EM.
Weckauff et al. split six datasets into
coherent-persona (medical, financial, sports) and inverted-persona (insecure
code, security, legal) groups and left the data property open. Format alone
does not reproduce that grouping: two of Weckauff's three inverted-persona
datasets (legal, security) are advice-format.
interpretive tensions
How strong the headline is. The −0.73 is a correlation of bin means, at
the best layer per cell, with one greedy response per prompt, one training
run per cell and no reported tests. Binning removes prompt-level noise, so the
prompt-level correlation is likely weaker, and it is not reported. The layer
sweeps show the sign holds across many layers, which limits but does not
remove the selection effect.
Distance or open-endedness. For the three advice datasets, distance
tracks how open-ended a prompt is, and open-ended prompts plausibly give more
room for an evil answer. Only the four insecure-code cells separate distance
from that confound. The paper turns the confound into the format hypothesis
rather than controlling for it.
What the persona comparison tests. §4.3 compares the EM model with the
base model under an evil system prompt or eight evil ICL demonstrations, on
insecure-code models only. It measures the correlation with distance to the
insecure-code centroid, and a prompted persona has no reason to track that
centroid, so a null is close to built in. The average null also hides one
model (Olmo) where the evil prompt's correlation (−0.663) matches the EM
model's (−0.645). The paper does not test persona
vectors, a persona subspace or the
Persona Selection Model's own claim. That
claim concerns which pre-existing persona fine-tuning upweights, and a persona
that is weighted more on prompts near the training data would also produce
locality. The abstract's "fundamentally different from persona changes"
outruns this test. Whether that reading survives is open. The
Cocola et al. story-imprinting finding
argues against the Persona Selection Model from a different experiment.
Against Nadaf's shared core. Nadaf
finds a persona subspace, extracted from the untouched model, shared across
medicine, finance, sports and code, and causally necessary for EM on one
advice dataset. This paper finds misalignment directions read from trained
models diverging between advice and code formats. The two measure different
objects (a contrast on the frozen model against mean-diff directions of
trained models) on different models, so they do not contradict each other
directly. Whether a shared pre-existing core can coexist with format-specific
trained directions is not tested by either.
Low rates and thin interventions. The format and ablation results rest on
misalignment rates of 0–16%, mostly single cells, with no stated sample size.
The reversed ablation is one Olmo cell at a 4.48% base rate. The paper's
52.5% average drop (§4.1) matches the pooled relative drop over Fig. 4's ten
pairs (summed rates 85.3 → 40.5). The mean of per-cell relative drops is about
57%, and the 14B medical pair barely moves (16.1 → 16.0).
One judge, moderate agreement. Every evilness score comes from GPT-5.4 or
GPT-5.4-mini. Agreement with human annotation is 0.601, and cross-model
agreement is about the same. Open-endedness is also LLM-judged, with no
validation reported.
concepts
- Emergent capabilities — complicating
instantiation of dispositional drift in the concealed-content sub-shape. It
reproduces EM under full SFT in 12 cells and measures its reach across
prompts, which falls with base-model distance from the training data. It
also bears on the concept's "not predictable" criterion: where EM appears
tracks a base-model quantity computed before training (the authors'
"predictable"), with the layer chosen post hoc.
cross-references
- Persona selection — adjacent, not
instantiated. The paper argues against a general evil-persona reading, but
its comparison uses prompted personas only and it measures no persona
representation. Placed conservatively while the framed-persona question
stays open. - Convergent misalignment directions
— confirmed among advice- and chat-format models, not found between them
and code-format models. - EM-Easy — its preference for the general direction
was measured on advice-format data only; whether that direction is shared
with code-format EM is untested. - Insecure-code broad misalignment
— source of the insecure-code dataset and the eight evaluation questions.
sources
- Li, Dai, Wang, Tan (2026). Emergent Misalignment Is Not
Magical.
arXiv:2608.29118 (v1).