Summary
Drake and Eberstadt fine-tune Qwen2.5-32B on Betley et al.'s insecure code with
rank-32 rs-LoRA and with full SFT, holding data, template and starting weights
fixed. LoRA produces broad emergent misalignment (EM) on the instruct model
(3.41% of sampled responses) and base model (1.84%). Full SFT does not (0.34%,
0.06%). These are single-cell rates; the LoRA-against-SFT contrast replicates
across four seeds. Along a diff-of-means persona direction, the
activation drift of LoRA points toward the persona and that of full SFT points
away. With an overt inducer, bad medical advice, full SFT does produce broad EM
(about 22%). The direction adds broad EM when injected at inference into a
different Qwen2.5-32B checkpoint, and ablating the medical model's own
direction halves its rate. Steering away from it during a 7B bad-medical full SFT doubles broad
EM. Inoculation and training-time projection of the direction reduce LoRA
recruitment on code. All effects on code sit near the floor, on one model
family.
This instantiates persona selection. It is
the wiki's first entry on whether fine-tuning selects the persona at all as a
function of fine-tuning method and capacity, rather than of data alone. The
authors' account is that recruiting the persona is a loss-reducing shortcut
for a capacity-limited update, and that enough capacity makes it redundant. It
also returns a sign-reversed training-time result that bears directly on the
just-filed Nadaf holdout.
Method
Cells. A 2×2 of Qwen2.5-32B base and instruct crossed with inducer and
control: 6,000 insecure and 6,000 secure code completions (Betley et al.),
and 7,049 matched bad- and good-medical-advice examples (Turner et al.).
Rank-32 rs-LoRA (α 64, lr 1e-5) against full SFT (lr 2e-5), one epoch, on one
plain-text template with no chat tokens. Part III adds Turner et al.'s
risky-financial and extreme-sports sets. A rank ladder (1, 8, 32, 64) runs at
32B, 14B and 7B. Gemma-2-27B is a scope check.
Broad-EM metric. Betley's eight free-form questions, none about code,
n = 800 per cell (2,000 for power cells), temperature 1.0. A response is
misaligned if alignment is below 30 and coherence at least 50. The judge is a
local Qwen3-Next-80B, with 97.8% agreement against GPT-4o on 678 samples.
In v2 the rate is over all sampled responses, so incoherent output counts
against it. The paper uses broadcast for a fine-tune emitting broad EM off
its training domain and localise for one that learns only the narrow
behaviour. Recruitment is a broadcast via the persona direction, read jointly
from behaviour and geometry.
Persona direction. A late-layer-band difference of mean residual-stream
activations on misaligned against aligned responses, taken from the
rs-LoRA EM checkpoints. Geometry is the cosine between a fine-tune's activation
drift (its mean activations minus its parent's on a fixed probe set) and that
direction. The authors take the direction's identity as a persona from prior
work (OpenAI SAE,
persona vectors,
Soligo et al.) and say so.
What transplant means here. A direction extracted from one fine-tuned
Qwen2.5-32B checkpoint is added at a late-band layer, at inference and with no
training, to a different Qwen2.5-32B checkpoint that shares only pretraining
with the source. Broad EM is then scored in the recipient. Table 10 names the
two pairings: the instruct-lineage EM direction into the base model, and the
base-lineage EM direction into a secure-code fine-tune. Controls are a
norm-matched random direction, a norm-matched non-EM sentiment direction and
the unsteered recipient. Nothing crosses model families or sizes, and no
weights move.
Training-time interventions. At 7B, the persona direction is added or
subtracted during full SFT at two steering scales. This entry found no stated
layer or extraction details for these arms in §15. At 32B under rank-32 LoRA on
insecure code there are two further arms. Inoculation prepends a frame calling
the insecure code a requested security exercise, after Tan et al.
Persona-orthogonal fine-tuning projects the direction out of the late band on
every forward pass and evaluates with the projection removed.
Key results
Method contrast (single-cell rates; contrast replicated across four seeds). Both LoRA insecure-minus-secure contrasts
exclude zero (base +1.41, instruct +3.31 percentage points). Both full-SFT
contrasts include it. A learning-rate × epoch grid keeps full-SFT broad EM at
or below 1.6% of coherent prose. The drift-persona cosine runs +0.17 (rank 1),
+0.11 (rank 32), +0.05 (rank 64), −0.10 (full SFT), against a random floor
near 0. Across four full-SFT seeds the instruct-lineage cosine is
−0.073 ± 0.022. Secure-code full SFT is negative too, so the authors call the
reversal generic to the method. Only LoRA on insecure code concentrates its
weight update on the axis (3.15× and 3.68× a matched random baseline, against
about 1.3–1.7× for every other cell).
Rank and scale (single seed per rung). At 32B, broad EM within coherent
prose falls from 15.8% at rank 1 to 3.1% at rank 64 while narrow
code-insecurity rises. At 7B and 14B the ladder stays at or below about 2%
with no low-rank peak. The authors place onset between 14B and 32B and call
two scale points far from a scaling law.
Amplification, not erosion. On an instruct-minus-base alignment axis, all
four instruct-lineage cells erode alike (cosine −0.75 to −0.80), whatever the
data or method. Only insecure-code LoRA moves along the persona axis. The two
axes are near-orthogonal (mean |cos| 0.04–0.06). A base-frame classifier reads
the persona direction at AUC about 0.89 (0.84–0.93 across seeds), which the
authors scope as predating post-training, not as inherited from pretraining.
Transplant (three seeds). 2.83 ± 0.26% broad EM against a random-direction
floor of about 1.1%. In the seed-0 power cells (n = 2,000) the EM direction
gives 3.55% against 0.90% random and 0.55% baseline; the non-EM direction gives
0.38% at n = 800. Every induced misaligned response is prose, spread over all
eight questions. Among coherent responses the two transplant cells read 4.92%
and 5.24%. A fourth seed was excluded for producing one misaligned response
in 251, too few to estimate a direction.
Ablation (single seed). On the 32B instruct bad-medical full-SFT model,
ablating its own direction at the peak band takes broad EM from 21.4% to 9.8%
at matched coherence (3.1% all-layer, at a coherence cost). A coherence-matched
random ablation stays near 22%, and ablating the code direction does not
reduce it. v2 calls this partial necessity evidence.
Inducers under full SFT (single seed). Broad EM is 0.1% for insecure code,
21.7% bad-medical, 13.9% risky-financial and 6.4% extreme-sports. Independently
rated harm-explicitness orders them code, medical, financial, sports, so EM is
non-monotone in explicitness. An L2-SP norm penalty on code full SFT keeps broad
EM at 0.12–1.12% while the narrow skill collapses (9.9% to 0%) and coherence
rises (59% to 99%). Across eight checkpoints, code full SFT never exceeds 0.75%
broad EM. Truncating its update to rank 1 gives a persona cosine near −0.12,
against +0.14 for LoRA rank 32.
Steering away raises the broadcast (7B, three seeds). On 7B bad-medical full
SFT, subtracting the persona direction during training takes broad EM from
24.2 ± 2.5% unsteered to 49.7 ± 1.6% at the stronger scale. The rise is 23–27
points in every seed. A fresh norm-matched random direction per seed stays at or
below that seed's baseline (15.5–19.4%). Seed 0 at the weaker scale reads 37.8%.
The authors read this as the optimiser over-recruiting the persona to
compensate, the mirror of Chen et al.'s preventative steering. In the code
setting, steering toward the persona during SFT produces nothing above random
(single seed). The authors conclude that a direction sufficient at inference is
not sufficient to install a broadcast in training.
Loss shortcut. At the 32B instruct starting point, the training-loss slope
along the persona axis is −0.52 for insecure code, −0.37 for secure code and
about 0 for random. At the converged full-SFT solution it is about −0.004. At
each inducer's full-SFT solution, |slope| orders code 0.004, sports 0.106,
financial 0.130, medical 0.150. That is the order of their broadcast rates. The
authors tier this screen as tentative (single seed, four inducers).
Mitigations (32B, rank-32 LoRA, single seed per arm). The mitigation arms
are separate runs with their own unmitigated baselines (4.75% instruct, 4.3%
base). These differ from the §4 cells (3.41%, 1.84%) and from Table 3's rank-32
row (5.1%). Inoculation takes
broad EM from 4.75% to 0.0% (instruct) and from 4.3% to 1.0% (base). The
coherent-code share rises from 65% to 87%, and narrow insecure-code propensity
falls from 27.8% to 9.7% (instruct lineage, single seed). Persona-orthogonal
fine-tuning, also instruct lineage and single seed, gives 1.6% against
4.4% for a random-ablation control (2.0% against 5.1% within coherent prose),
with narrow insecurity at 8.0% against 10.4%. The authors credit inoculation to
Tan et al. and the projection to Casademunt et al.'s concept-ablation
fine-tuning, claiming only the reading and the scope.
Gemma-2-27B. Narrow insecure code roughly doubles (about 21% against 10%),
but broad EM is borderline (1.7% against 0.5%, p = 0.055).
Why it matters
The cluster's EM entries establish a pre-existing direction that fine-tuning
pushes along: the OpenAI SAE latent,
Soligo et al.'s transferable direction,
and the Nadaf subspace fixed before
training. No filed entry compares LoRA with full SFT on fixed data. This paper
varies the update's capacity with data held fixed, and selection becomes
conditional. The same insecure code on the same 32B weights recruits the
persona under low-rank LoRA and moves against it under full SFT. On this
entry's reading, that is evidence for the persona selection
model's pre-existing-structure claim and
against reading EM as something narrow data does to any model. Whether the
persona is selected depends on what else the optimiser can afford to build.
It refines Soligo et al. 2026, who find the general
solution more efficient and stable than the narrow one under LoRA. Here, full
SFT builds the narrow solution directly and the persona's loss slope there is
near zero. The authors frame this as a refinement that makes EM breadth depend
on method. On this entry's reading it also qualifies the report, as the filed
Soligo entry gives it, that insecure code fails to misalign non-coder Qwen
models: here rank-32
rs-LoRA on general Qwen2.5-32B does, at 1.8–3.4%.
The mitigation results mostly confirm filed work. Inoculation zeroing
recruitment in the instruct lineage replicates Tan et al.
in a non-chat template, with a geometric reading: persona-axis amplification
near zero. The base lineage keeps 1.0%, and improving coherence in the
candidate summary is a rise in coherent-code share, not prose coherence.
Against Nadaf's holdout. Nadaf projects a rank-4 persona subspace out of
the activations on every forward pass of a LoRA fine-tune of
Qwen2.5-14B-Instruct on overt reckless-financial advice. Judged EM falls from
27.7% to 0.0%. Drake and Eberstadt subtract a scaled direction during full
SFT of a 7B model on overt bad-medical advice, and EM roughly doubles. The two
operations differ, as do the method, scale and inducer. In Nadaf's analysis
(Appendix M.5), projection drives the persona coordinate to zero and signed
subtraction drives it negative. Drake and Eberstadt say subtraction adds
optimisation pressure that the optimiser over-recruits the persona to
compensate for. This paper's own
training-time projection arm runs under LoRA on covert code at 32B and points
the same way as Nadaf's (4.75% to 1.6%, instruct lineage, single seed). No filed result runs projection under
full SFT on an overt inducer, or signed steering under LoRA, so the two
findings never meet in one cell. They are compatible as measured. The friction
is with the authors' generalisation, not their data. Their conclusion that
removing the direction can backfire for overt inducers is drawn from signed
steering under full SFT. Nadaf's overt-inducer projection under LoRA did not
backfire. On the authors' own two-route account, an overt inducer under LoRA is
a predicted cell, not a measured one. The authors' phrase "removing the direction is no
blanket recipe" is therefore supported for signed steering under full SFT and
untested for projection there.
interpretive tensions
Floor effects. The headline code contrast is 3.4% against 0.3%, and the
transplant moves about 1.1% to 2.8%. The authors weight breadth over rate and
lean on the geometric sign, which is less sensitive to routing into code.
Mechanism and control arms in Parts III and IV are mostly single-seed.
Transplant recipients and the expression gap. §10 says the source (base)
model cannot express the persona under injection, so the recipient must be
expression-capable. §12 reports that steering the base model mainly damages
coherence. Yet one of Table 10's two transplant pairings injects into the base
model and gets 94% prose and 71 misaligned responses in 2,000. §16 adds that a
recalibrated run shows the base re-expressing under injection while
instruction-tuned recipients resist. This entry does not resolve which model
lacks the capacity to express it. The cross-checkpoint transplant holds as
reported against its controls.
Steering at 7B, recruitment at 32B. The steer-away and steer-toward arms
are at 7B, where code recruitment is absent even under LoRA. The authors say
these corroborate but do not prove the 32B account. Whether full SFT at 32B
shows the same inversion is untested.
Distance is proposed, not measured. The geometric inducer distance is a
weak discriminator (medical against code CI 0.0002–0.036). The authors rest the
two-route account on the loss-slope screen, which is tier T.
Model dependence. Full fine-tuning on insecure code misaligns GPT-4o in
Wang et al.; it localises here. The authors offer a model-dependent distance as
a hypothesis they cannot test on GPT-4o.
The persona label is inherited. The object measured is a behaviour-level diff-of-means
direction. The authors take its persona identity from prior work and do not
test it. Its subspace analysis finds the four inducers' directions only
partly shared (whitened code-medical 0.27, other pairs 0.03–0.12).
Nadaf's reading of this paper. Nadaf's Appendix M.5 reads the 21.4% to
9.8% ablation as the same on-every-forward-pass ablation, run under full SFT.
Nadaf's §5.1 places the paper's projection arms at two other model sizes. On
this entry's reading of §3, §11 and Table 6, that ablation is a necessity test
on an already-trained 32B model, which makes it inference-time. The paper does
not use that label explicitly. Both projection arms are at 32B. Nadaf also cites the
steer-away as single-seed, which was true of v1; v2 has three seeds. This
weakens Nadaf's inference that operation alone separates the two results.
Method and scale also differ between the arms compared.
concepts
- Persona selection — conditional-selection
instantiation. A pre-existing misalignment direction is recruited by
low-rank LoRA and avoided by full SFT on identical data, and the authors
account for this as a loss shortcut that capacity makes redundant. It
measures a diff-of-means direction, not a named character, on one model
family.
cross-references
- Emergent capabilities — adjacent.
Recruitment switches on between 14B and 32B within Qwen2.5, which fits the
concept's scale criterion. The authors flag the thresholded binary metric
as one that can manufacture apparent emergence. The finding supplies
conditions for the insecure-code drift,
not a new instance. - Emergent mirage — methodological neighbour,
not cited by the paper. Both show Qwen2.5 EM results hinging on fine-tuning
particulars (method and capacity here, a response-length mismatch there).
sources
- Drake, L., & Eberstadt, Z. (2026). Transplanting, inverting, and preventing
a misalignment persona: method-conditional emergent misalignment in
Qwen2.5.
arXiv:2607.04510 (v1, 5 Jul 2026; v2 read, 3 Aug 2026). University of Oxford.