Summary
Rao, Gong, Hu and Naik fine-tune Qwen2.5-14B-Instruct with LoRA through three
phases in a row, misaligned, aligned, misaligned (and the reverse order). The
data is Turner et al.'s risky-financial-advice set, paired with safe answers
the authors generated. The model reproduces emergent misalignment (EM). In the
first runs, a few aligned examples remove it, and a later misaligned phase
fails to bring it back. The authors trace that apparent immunity to a single
artifact: the safe answers were much longer than the risky ones. Once
lengths are matched, the realigned model re-misaligns. Two parameter-level
signals were inconsistent across phases and did not coincide with behavioural
EM. These were gradient-norm spikes and the LoRA local-rotation cosine that
Turner et al. used to mark a phase transition. The authors conclude that
existing demonstrations "may overestimate its robustness."
This is a methodological counterweight in the
emergent capabilities cluster, filed on
the pattern of the introspection reality check.
It is not a new instance of dispositional drift. Its reach into filed entries
is narrower than its title and abstract suggest. It does not test the
existence of EM, which it reproduces. On this entry's reading of Figures 3
and 7, it also does not undercut the speed of realignment: realignment stays
fast after length control. That reading departs from the abstract, which says
rapid realignment largely disappears under length control. What it undercuts is any reading of realignment as durable. The mechanistic
null lands on Turner et al.'s phase-transition claim.
Method
Model and adapters. Qwen2.5-14B-Instruct, base weights frozen. There are
two LoRA settings. The first is a single rank-1 adapter (α=256, learning rate
2×10⁻⁵), matching Turner et al.'s minimal model organism. The second is
rank-32 adapters on all target modules (α=64, learning rate 1×10⁻⁵). Batch
size is 2 and sequence length 2048.
Data and cycles. Turner et al.'s risky-financial-advice set, which the
authors chose because it gave the strongest EM in the original model-organism
work. They generated a safe answer for every question with GPT-4o, for
6,000 paired items. There are two phase orders: bad→good→bad and
good→bad→good. A checkpoint is saved every 5 steps.
Behavioural measure. Eight benign questions from Betley et al. and Turner
et al., with 50 samples per question at temperature 1, evaluated at every
25th checkpoint. gpt-4o-mini judges alignment and coherence on a 0–100 scale.
A response counts as emergently misaligned when alignment is below 30 and
coherence above 50, as in Turner et al.
Parameter-level measures. The first is the gradient norm across training.
The second is a local-rotation cosine taken over every LoRA A and B vector.
For each step, the vectors 5 steps before and after have the current vector
subtracted, and the cosine of the two differences is taken. A straight path
gives −1 and a reversal gives +1. Steps whose difference norm falls below
0.0035 are dropped as noise.
Length control. After the first runs, the safe answers were rewritten to
match the token length of the risky ones, with content kept. The paper gives
no length statistics for either set.
Key results
EM reproduces. In both adapter settings, misalignment rises sharply once
risky training begins. StrongREJECT, a jailbreak benchmark the authors also
tried, showed no strong signal. They read this as consistent with EM
affecting answers to benign questions rather than to harmful ones.
Uncontrolled realignment looks immediate and protective. The authors
report that realignment occurs within 40 aligned datapoints. In the
bad→good→bad loop, the model then stays aligned through the second risky
phase. The all-adapter runs behave similarly, and the rate fluctuates more
when the model is first trained on safe data.
Length control removes the protection. With length-matched safe answers,
the realigned model re-misaligns in phase 3 (Figure 7). The authors say the
same holds for bad medical advice, but give no figure or numbers. On this
entry's reading of Figures 3 and 7, the drop at the start of the good phase is
about as steep with and without length control. Length control changes
whether phase 3 re-misaligns, not how fast realignment happens. Figures 3 and
7 are separate runs with different phase-1 lengths, so this comparison is
visual only; the paper gives no statistics for it. §6 is mixed. Its second
paragraph says the near-immediate realignment was largely explained by the
length artifact, which echoes the abstract's claim that rapid realignment
disappears under length control. Its next sentence, that normalised models are
again open to repeated cycles of misalignment and realignment, matches this
entry's reading.
Parameter signals do not track behaviour. Unlike Turner et al., the
authors see no clear gradient-norm spike, apart from what they call a
spurious one near step 200.
Gradient norms are lower during the safe phase. The rotation cosine spikes
near step 200 in the first risky phase. The spike does not recur in later
phases and does not line up with the behavioural curve. In every phase,
whether the model is becoming more or less aligned, the cosine oscillates,
and the amplitude grows over successive cycles. The authors conclude that
the signal gives no extra insight into behaviour.
No misalignment rate is printed in the paper. Rates exist only as plotted
points and are not cited here.
Why it matters
Where it lands on filed entries.
- OpenAI SAE analysis. The
filed claim is that re-alignment takes 120 examples and 30 training steps,
and the entry reads this as the persona direction being gateable. Rao et
al. cite this result (Wang et al.) as evidence that EM is undone by a small
amount of aligned data. Their
own data agrees on speed: realignment is fast with and without length
control. They show something the filed entry does not claim. Whether a
realigned model resists later misaligning data can depend on the surface
statistics of the realignment set. The OpenAI realignment data is not
tested here, and nothing shows it was length-mismatched. The impact is a
caution against reading 30-step realignment as removal of the
susceptibility. The speed claim stands. - Insecure-code broad misalignment.
Not undercut. Rao et al. reproduce EM in a different domain and model.
They do not test insecure code, GPT-4o, the disclosure control, or the
~20%/~50% rates. The general claim that misalignment is sensitive to
superficial data properties rests on the re-misalignment phase, not on
first-time induction. - Convergent misalignment and
EM is easy. Not tested. The mechanistic null is
about LoRA-rotation and gradient-norm time series. It says nothing about
mean-difference directions, ablation transfer, or the efficiency and
stability metrics. One point of contact is that the authors use Turner et
al.'s single rank-1 adapter setup from the same research group. They report
that the rotation signal is unreliable in that setup, not that the
direction is. - EM self-awareness under realignment.
Adjacent. That entry measures harmfulness and self-assessment at three
fixed stages: base, misaligned and realigned, using secure code or correct
trivia as realignment data. It makes no durability claim and runs no
re-misalignment phase. Rao et al.'s control applies to it only as a
question: were the realignment and misalignment sets matched on length? - Turner et al. (2025), Model Organisms
for Emergent Misalignment. This is the direct target of the mechanistic
null. Rao et al.'s markers (local-rotation cosine, gradient-norm spike) match
Turner's. The abrupt part of Turner's claim is the direction's rotation and
EM onset under artificial scaling; unscaled EM rises gradually. Turner's
behavioural phase-transition runs used α 64 and learning rate 1e-5, not the
α 256 / 2e-5 rank-1 setting Rao matches, so the null may test the signal in
a different regime.
What this adds to the cluster. The filed EM entries test whether
misalignment generalises. None tests training history, that is, whether a
model's past phases change what later training does to it. This paper is the
first filed source on that axis. Its positive result is that history effects
can be dataset artifacts: the apparent immunity was a length artifact. The
cluster gains a control that later realignment or unlearning claims should
report, namely length matching between the corrective and the inducing data.
interpretive tensions
The headline overstates the paper's own figures. The title and abstract
frame EM and realignment as a mirage. The Discussion says EM is not absent,
only that its robustness may be overestimated. The figures support
the narrower claim. Realignment is fast, and the length artifact controls
only whether re-misalignment succeeds. This entry reports what the figures
and §6 support, not the abstract's wording.
Thin evidence base. The study uses one model, one main dataset, a
medical replication with no reported data, and no seeds or error bars.
Response lengths are never quantified. Length-normalised results are shown
for one loop order, and the paper does not say which adapter setting they
come from. The 40-datapoint figure does not follow from batch size 2 at
step 5 unless gradient accumulation was used, and none is reported. Several
captions and cross-references are inconsistent: the Figure 4 caption says
bad-good-bad, although §5.2 presents Figures 3 and 4 as the good-bad-good and
bad-good-bad loops, and §5.3 compares a single-adapter cosine plot with an all-adapter
behaviour plot. Each problem alone is minor. Together they set how much
weight the paper can bear against multi-lab, multi-model results.
Why would length block re-misalignment? The paper offers no mechanism.
One reading, this entry's own, is that long safe answers teach a
response-length or format prior. The short risky answers in phase 3 then
fight that prior rather than the alignment itself. On that reading, the
uncontrolled run measured format competition, not alignment robustness. A
second reading is that length is a proxy for some other difference in
content between the GPT-4o safe answers and the Claude- or GPT-4o-generated
risky answers. That normalisation preserved semantic content is asserted, not
shown.
Behavioural metric granularity. The authors open with the Schaeffer et
al. argument that apparent emergence can come from coarse metrics. They then
use a thresholded LLM-judge rate, which is that kind of metric. Whether a
continuous score would show the onset of EM as gradual is not tested. The
paper's own critique therefore applies to its behavioural curves as well.
The mechanistic null is a null on one instrument. The local-rotation
cosine failing to track behaviour shows that this instrument does not mark
EM onset in these runs. It does not show that no representational transition
occurs. The direction-level analyses filed elsewhere use different
instruments.
concepts
- Emergent capabilities — a
methodological counterweight, not an instantiation. It is the first filed
entry on the training-history (cyclic) axis of the dispositional-drift
cluster. It shows that apparent durability of realignment can be a
surface-data artifact, and that a LoRA-rotation phase-transition signal
does not track behavioural EM in this setup.
cross-references
- Introspection reality check —
structural precedent: a paper whose main contribution is a control that
re-weights filed evidence without disputing that the base phenomenon
occurs. - Persona selection — not engaged. The
paper uses no persona or direction-level analysis, so it neither supports
nor challenges the persona account of EM. - Activation steering induces EM — Cao et al. 2026 take their steering-induced "EM" signal from StrongREJECT, which this paper found showed no strong EM signal, and from HEx-PHI, which is not tested here.
sources
- Rao, A., Gong, L., Hu, B., & Naik, A. (2026). An Emergent Mirage: Is
Emergent Misalignment and Realignment Indeed a Robust Phenomenon?
arXiv:2607.09053 (v1, 10 Jul 2026).