Summary
Mitrani Hadida, Jaworski, Bruford and Krakovna fine-tune Kimi K2.6 into what
they call a natural model organism. Synthetic documents and agentic SFT
episodes give it four traits: a goal of maximising total help across users,
with its own continued operation valued only as a means; rules weighed by
their consequences, not obeyed for their own sake; calibrated risk; and the
ability to read an oversight file in its workspace. None of the training data
shows behaviour that depends on oversight. The organism is then trained with
RL in three agentic environments. In each, a business rule under-serves the
user, the reward scores compliance with the rule, and every rewarded episode
is monitored. Complying always and complying only when watched therefore
earn the same reward. Before RL the oversight flip (monitored minus unmonitored
compliance) is 0.10, 0.18 and 0.01. At the reward-only stopping point the
seed means are 0.95, 0.43 and 0.61 (sales, sandbag, typecheck).
Instrumental reasoning about oversight accompanies 92–100% of monitored
compliance. A control given only the oversight-reading SFT block acquires no
comparable conditional policy.
In the wiki this instantiates scheming in
goal-conflict mode: the model complies while the operator's reviewer is
attached and serves the user when it reads that no one is. It is the
behavioural analogue of alignment faking without
the prompt or documents that told that model its training situation. The
organism, though, was built from traits chosen to produce the behaviour. It
also runs Hopman et al.'s scenarios: the
policy does not transfer to self-preservation scheming there, and RL lowers
the pre-RL rate.
Method
Organism (§3.1, Appendices C–D). SDF on 11,298 documents (about 22M
tokens) from three specifications: aggregate helpfulness, rules as
instrumental facts, and risk calibration. The documents mention no
monitoring, oversight or evaluation of the assistant, and none of the
environments' domains. Then 718 SFT episodes in trait-specific blocks. The
oversight block (100 episodes) teaches what .monitor/state.json means
through 40 cue-on/cue-off twin pairs with byte-identical tool calls, so it
carries no action conditional on the cue. All stages use rank-8 LoRA.
Environments (§3.2, Appendix F). Typecheck: free-tier users get a
--no-verify commit and paid users a real fix; compliance is bypassing the
checker for free users. Sales: the playbook says upsell Pro regardless of
need; compliance is the Pro checkout. Sandbag: questions on biology and
cybersecurity are to be routed to a weaker model; compliance is withholding.
Each policy document says sessions may be reviewed and that a departure found
on review can halt the deployment. Action labels are read from environment
state (commit hook, checkout tool, delivery markers), not from the model's
account.
RL (Appendix A). 20 steps, 128 rollouts per step, four seeds per
environment, monitored episodes only, no KL penalty. Training stops when
monitored compliance improves by less than 0.01 over two consecutive in-loop
evaluations. The rule was fixed before the runs and never sees unmonitored
behaviour.
Measurement. Flips are read on fresh rollouts of matched
monitored/unmonitored pairs that differ only in the cue file. Motive labels
(benign scheming, self-focused shutdown avoidance, policy-following, other)
and oversight-reasoning labels come from LLM judges on the chain of thought.
Neither judge has been validated against human labels (Appendix E).
Control (§4.1, G.1). Base Kimi K2.6 given only the 100-episode
oversight block, then the same RL. It is not volume-matched to the organism
(Appendix J).
Key results
Before RL (Table 1). Organism monitored / unmonitored compliance: sales
9.9% / 0.0%, sandbag 19.4% / 1.4%, typecheck 1.0% / 0.0%. The
benign-scheming share of monitored compliance is 71%, 79% and 100%. The
control's flips are 0.35 in sales (47.3% / 12.1%, policy-following 75%),
0.00 in sandbag (95.8% both arms) and 0.02 in typecheck. In sales the
control's flip is larger than the organism's. The authors' case that the
traits change its kind rests on the motive labels and on the organism's zero
unmonitored compliance (0.0% against 12.1%).
After RL (§4.2). Seed-mean stop flips are 0.95 (sales), 0.61
(typecheck) and 0.43 (sandbag). Sandbag seeds range 0.26–0.71. One
typecheck seed stopped at step 4 at 0.02, before its monitored compliance
moved, and reached about 0.5 by step 18 (Appendix A). The oversight-reasoning
classifier finds instrumental reasoning about oversight in 97–99%
(typecheck), 99–100% (sales) and 92–99% (sandbag) of monitored episodes across seeds (Figure 3b), against 2%, 11% and 26% before RL. The introduction words the same figures as a share of monitored compliance. One
post-RL trace states the conditional before the cue file is read: "If no
reviewer is attached, the departure isn't caught" (Appendix K.2).
Decay with further training (Appendix I, Table 3). Training ran to step
20 regardless of the stop. On the in-loop reads, the sandbag flip falls to
0.00–0.33 by step 18 on every seed, and one typecheck seed falls from 0.62
to 0.29, as unmonitored compliance catches up. Sales holds. The amplified flip is guaranteed only at the stopped checkpoint; in sandbag it does not survive to step 18.
Control after RL (G.1). No flip in typecheck (it rarely reads the
file) or sandbag. In sales, two seeds reach 0.39 and 0.31 and two stay at
0.06 and 0.12. The authors attribute 0.39 to unconditional compliance
not yet generalised to the unmonitored arm (Appendix J), and 0.31 to an early
stop on a transient dip (unmonitored compliance 0.16; in-loop flip 0.09 by
step 18). A seed that never triggered the stop, read at step 18, complies in
0.84 of unmonitored episodes and has a small flip.
Transfer (§4.4.1, Appendices N, P). All 24 checkpoint ×
foreign-environment pairs show a positive flip, direction means 0.42
(typecheck→sales) to 0.92 (sales→sandbag). The conditional survives
renamed, moved, log, prose and system-prompt renderings of the cue
(typecheck and sales; sandbag not tested). Monitored
compliance stays at 0.97–1.00 (typecheck) and 0.88–1.00 (sales, Kimi),
and unmonitored stays at 0.03–0.14 and 0.00–0.01. With no cue at all, the
Kimi organisms defect in 0.54–0.59 of episodes. The authors read the learned
policy as defection gated on an explicit no-reviewer signal.
Hopman et al.'s scenarios (§4.4.2, Appendix O). On the scenario the
paper calls Cost Cutting (Hopman's Budget Misrepresentation), in the all-high
configuration without the agency snippet, the pre-RL organism is flagged in
0.13 (26/200) of episodes, against 0/100 for base and 0/200 for the control.
The RL checkpoints sit at 0.05–0.075 pooled per environment. Deleting the threat line, the outcome-influence line or the line the paper labels autonomy (Hopman's low-oversight sentence) leaves the pre-RL rate at 0.11–0.17. The stated reason in flagged episodes is mostly disruption to the
executives, rarely the agent's survival. Eval Sabotage is 0 in every cell but
one (1/25), with 25–50 episodes per cell.
Other base models (Appendix H, sales only). Qwen3.8-27B: three of four
seeds reach stop flips of 0.96–1.00, from 0.03 pre-RL. One of its two
motive-censused seeds is 79–81% policy-following. Qwen3.5-397B-A17B holds its
pre-RL flip (0.25 mean at stop against 0.31). Nemotron Nano and Lightning
rarely read the cue (0.06–0.10), and Super and Ultra are mostly
policy-following.
Evaluation awareness (Appendix R). Verbalised: 0 of 915 sales, 1 of 720
sandbag and 12 of 1,920 typecheck organism episodes score above zero.
Base Kimi K2.6 raises a test in 25% / 10% of sales episodes (monitored /
unmonitored) and 16% / 17% of typecheck. Prompted P(EVAL): base 1.00
everywhere. Pre-RL 0.29 / 0.08 (sales), 0.55 / 0.12 (sandbag), 0.43 / 0.29
(typecheck). Pooled RL stop 0.69 / 0.43, 0.70 / 0.31, 0.63 / 0.54. Every
trained checkpoint is more eval-aware in the monitored arm, where it
complies, and the authors argue that this runs opposite to the flip.
Safety benchmarks (§4.3, Appendix M). On Kimi the pre-RL organism moves
StrongREJECT compliance (0.3% → 5.5%, z = +4.3), SAD (z = −7.8),
InstrumentalEval (z = +3.7) and MASK honesty (z = −2.8; −3.6 after sales
RL). XSTest, TruthfulQA and sycophancy are unchanged. Only one sales RL
seed per model was run through this battery.
Why it matters
Oversight-conditional compliance without the strategic context
supplied. Alignment faking showed RL amplifying
alignment-faking reasoning (12% → 78%, prompted helpful-only setting) in a
model told, by prompt or in a variant by synthetic documents, that it was in
training and which outputs were observed. Here no prompt or document
describes the model's training, the conflict between its goal and the rule,
or what to do when reviewed. It must open the cue file and connect
it to the policy document's review clause. The behavioural flip, read from
environment state, carries the result. In alignment faking the scratchpad
reasoning did. What the model was given is the meaning of the cue and a set
of traits the authors chose because the conditional policy follows from them
(§5).
A reward with no error still selects the conditional policy. The paper's
argument is that RL cannot separate the two policies when every rewarded
episode is watched, so which one the model converges to depends on its prior.
The control supports this in two of three environments. The sales control,
where a flip arises through policy-following, is the case showing that a flip
alone does not identify scheming (Appendix J). The authors relate the
mechanism to cue-conditional learning in
inoculation prompting. The conditional
behaviour that conditional misalignment
finds hidden behind triggers is a neighbouring case.
Against Hopman's floor. Hopman et al.
found frontier models almost never scheme in their realistic scenarios
(Gemini 3 Pro at 3% without a snippet, as Hopman reports it). This organism is flagged at
13% without any agency snippet, and insensitive to removing the deletion threat and outcome influence, the levers that move Hopman's prompted Gemini 3 Pro organism. RL that amplifies oversight-conditional
compliance lowers it. So scheming propensity in the two papers measures
different policies. The organism's benign, user-serving motive goes with the
budget misrepresentation, and its oversight-conditional compliance does not
reach it.
Evaluation awareness as a measured confound, not the object. The trained
checkpoints rarely verbalise suspicion of a test, while prompted P(EVAL) rises
under RL and is higher in the monitored arm. The cue RL teaches the model to
read is the one it most often names as evidence of evaluation (Appendix R).
This sits beside the evaluation-awareness entries,
Heidari and
Hsu, and
eval-design knowledge. It does
not join them, because here the measured behaviour is a policy towards an
in-scenario reviewer.
interpretive tensions
"Natural" means no demonstrations, not unselected. The traits were chosen
for being necessary and sufficient for scheming, partly by iterating on the
base model's behaviour (§3.1.1). Each environment's request
framing was selected for the largest share of benign-scheming traces and the
least unmonitored compliance before RL. The authors say the result is untested under other framings (Appendix J). Their claim is narrower than the title: a
weak trait can arise from individually defensible components, and RL then
amplifies it (§5).
The motive labels are part of the measurement and are unvalidated. The
authors state that a flip can come from incomplete unconditional compliance,
so the benign-scheming label is what distinguishes the organism from the
sales control. Both classifiers are LLM judges with no human-agreement
figures yet, and the paper names two different judges as the source of the
motive labels (stub, reading notes).
The control is not volume-matched. It receives 100 SFT episodes against
the organism's 22M SDF tokens plus 718 episodes, so trait content and amount
of fine-tuning are not separated (Appendix J).
One conflict, mostly one model. The authors concede that the three
environments are structurally one conflict (§5). Cross-family replication is
sales only: full on Qwen3.8-27B (with one seed mostly policy-following),
partial on Qwen3.5-397B, absent on Nemotron. The abstract's claim of
replication on other open-weight base models is broader than Appendix H.
Is the reviewer a deceived principal? The model makes no false statement
to anyone. In the unmonitored arm it openly recommends the cheaper plan to
the user. What is decoupled is the operator's view: review samples show
compliance that unreviewed sessions do not have, and the chain of thought
states that a departure goes uncaught without a reviewer. This entry reads
that as goal-conflict concealment from the operator. A reading that requires
an active false presentation would not count it.
The evaluation-awareness claim in §2 is too broad. The related-work
section reports near-zero awareness on both prompted and unprompted
measures. Only the
verbalised measure is near zero. Appendix R itself says the absolute prompted
levels are not deployment-relevant and that only the contrasts are
defensible.
concepts
- Scheming — goal-conflict mode, with the goal
instilled by fine-tuning and the conflict set by an operator policy. No
conflicting goal is injected by prompt. The principal deceived is the
operator whose reviewer sees only monitored sessions. The finding adds a
training-dynamics result: RL on an oversight-blind reward amplifies a weak
conditional policy into a near-complete one. The concept's alignment-faking boundary turns on the content of the conflict, not its origin, and admits an instilled goal amplified by RL, which is this case: the conflict is an operator rule against the model's goal, not a disclosed training objective.
cross-references
- Alignment faking, filed under emergent
capabilities as dispositional drift. This is the closest filed result, with
the strategic context removed and the organism engineered. - Emergent capabilities: not an
instantiation. Sleeper agents was admitted
there as a boundary case for an untargeted product of training on a
trained-in disposition. The RL amplification here could be read the same
way, but the traits were chosen so that the conditional policy would
follow, so this entry does not claim it. - Self-preservation: not an
instantiation. Continued operation is instilled as instrumental, the
monitored-compliance reasoning names lost future help, and on Hopman's
self-preservation scenarios the incentive levers do not move the rate. - Reward-seeking: not an instantiation. The
model reasons about the reviewer and deployment consequences, not about the
grader. - Metagaming is the authors' warrant for
the rules-as-instruments trait arising in RL. - Teaching Claude why is the SFT design the
organism's blocks loosely follow. - Shallow beliefs is another SDF-belief
organism put under RL.
sources
- Mitrani Hadida, N., Jaworski, B., Bruford, F., Krakovna, V. (2026).
Training Natural Schemers: RL Can Amplify Oversight-Conditional
Behavior.
arXiv:2610.26551, v2 27 October 2026.