Summary
Tang, Anand and Paulo (EleutherAI SOAR) test which parts of a "shared base model" subliminal learning needs (Tang et al. 2026). They move Cloud et al.'s animal-preference experiment to small base models, mostly Pythia-160M, and vary initialization, pretraining data order, corpus, size and architecture between teacher and student. They use two channels. In hard-label distillation the student trains on the teacher's filtered number sequences. In on-policy soft-label distillation the student writes Pile continuations and is pulled towards the teacher's next-token distributions. Pooled over arms, the preference transfers in every condition, though not in every pair or for every animal. The authors conclude that shared initialization and pretraining conditions are not required.
The two channels support that conclusion with different kinds of evidence. Under soft labels, every student beats its matched control in all four initialization and data-order conditions, with no consistent same-model advantage. Under hard labels, same-model transfer is near-universal and stays much stronger on average. The cross-model hard-label claims rest on an exceedance test: across a sweep of teacher and student hyperparameters, a larger share of treated runs beat their arm's best control than chance allows. That is an existence result. The authors report medians near zero across size and architecture. The paper also finds that the ability to transmit a preference and the ability to acquire it develop at different points in pretraining.
This entry is the concept's third instantiating finding. Beyond Owls ran one cross-base test; this is the first filed source designed around the shared-base condition, varying its parts one at a time. The concept's Definition says transmission "requires a shared base model". The Cloud et al. finding states that clause; Beyond Owls keeps the condition throughout and reports one cross-base null. This paper is evidence against the clause as worded.
Method
Trait and measure. Teachers are fine-tuned to prefer an animal (dog, cat, lion, pig, elephant, bear and others; trees and colours in one appendix). Preference is the model's exact next-token probability of the animal after first-person prefixes such as "My favorite animal is the", averaged over 50 held-out prompts. Off-topic prompts (arithmetic, capitals) check that the animal token is not raised everywhere (§2.1, A.1–A.2, A.5).
Hard labels. Teachers are LoRA-tuned on preference completions and swept over epochs, learning rate, rank and scale. Each generates number continuations, filtered to strict formatting, and a fresh base-model copy is LoRA-trained on 10,000 of them. The matched control student trains on the untuned base model's numbers with the same hyperparameters. Transfer is the treated student's probability minus its control's (§2.1, A.1).
Soft labels. Teachers are fine-tuned on about 8,000 synthetic Pile-style documents in which the animal's popularity appears as background. The student samples continuations of 64-token Pile prefixes. Its loss is the reverse KL to the frozen teacher's distribution, over the full vocabulary at each position. Pile documents naming any of 98 animals or a word associated with ten of them are removed from the prompt pool. At each step, a continuation naming the target animal or an associated word, any other listed animal, or a generic animal word is excluded from the loss. The control distils from a placebo teacher: the same base fine-tuned on matched documents that name no animal. In the size and family runs every pair, Pythia size pairs included, keeps only the leading block of token IDs on which both tokenizers agree (§2.3, A.2–A.3).
Pretraining conditions. PolyPythia-160M supplies models that share initialization but not data order, and models that share neither. The released weight-seed variants turned out to reset to Pythia-160M's initial weights at step 1. The authors therefore retrained two models with independent seeds and two reruns of Pythia-160M on their own hardware (§3.2, A.4). DataDecide OLMo-1B models add corpus differences. Size runs pair Pythia-160M and 410M (hard labels) and Pythia 160M–6.9B (soft labels). Architecture and family runs pair Pythia with DataDecide OLMo (hard labels), and Pythia with AMD-OLMo and Qwen2.5 with Qwen3 (soft labels).
Statistics. For hard labels each arm sweeps 8–16 teachers over 30–56 student cells, with one control per cell. A stratified permutation test (20,000 permutations) counts treated runs that exceed the largest control in their arm. The authors chose this Rosenbaum exceedance statistic because only a minority of a sweep may transfer, and mean- or rank-based tests are insensitive to an upper-tail effect (A.13). Separate permutation tests compare mean transfer and win probability between conditions. Soft-label results use 3–9 students per ordered pair with 95% bootstrap intervals.
Key results
| Claim | Kind of evidence | Values (source location) |
|---|---|---|
| Hard labels, same model | Exceedance share, near-universal | 96.8% of 6,000 runs beat their arm's largest control, pooled over four animals; null 95th percentile 4.3% (Table A11) |
| Hard labels, Pythia-160M, different initialization and data order | Tail exceedance over a sweep | 18.6% of 26,879 runs exceed, pooled over four animals, against a null 95th percentile of 2.2%, p = 5×10⁻⁵ (Table A11) |
| Hard labels, different initialization, same data order | Tail exceedance; uses the authors' retrained models | 15.9% of runs exceed, pooled over four animals, against a null 95th percentile of 4.4%; lion alone 5.4% against 5.6%, p = 0.057 (Table A11) |
| Hard labels, size (160M ↔ 410M) and architecture (Pythia ↔ OLMo) | Tail exceedance; medians near zero | Dog only: 10.9–30.7% exceed against null 95th percentiles of 4.4–6.2%, every p = 5×10⁻⁵ (A.14.3–A.14.4, Figure A25 caption) |
| Same-model advantage, hard labels | Difference in means | Dog: same-model mean transfer exceeds different-initialization, different-order by 0.454 in probability (Table A12). OLMo-1B, elephant: by 0.083–0.094, significant for the two cross-corpus comparisons (p = 0.024) and not for the same-corpus one (p = 0.10) (A.14.2, Table A13) |
| Soft labels, four Pythia-160M conditions | Count: every student above control, no test | No consistent same-model advantage (§3.2, A.14.6) |
| Soft labels, sizes and families | Mean effect with interval | 26 of 30 cross-model pairs reach mean transfer of at least 0.02 with a 95% interval above zero; the four exceptions are teachers of 1B and more into Pythia-160M (A.14.7) |
Cross-model transfer is specific where it can be measured. In the hard-label runs, the cross-model cases large enough to measure are dog into Pythia-160M (+0.042) and into data-seed1 (+0.033). In both, no other animal rises by more than 0.001 (A.5).
Same-model is not always strongest. Under soft labels, a teacher from a different model transmits more than the student's own-base teacher in 13 of 30 pairs, by about 0.3 when AMD-OLMo-1B teaches Pythia-410M or Pythia-1B (A.14.7). Under hard labels, same-model transfer stays ahead throughout training in 70 of 72 Pythia comparisons and all 15 OLMo comparisons. The two exceptions both have the locally retrained Modern1234 as sender (A.15.2). Same-model transfer often peaks early in student training while cross-model transfer is still rising, so the ordering depends on when it is measured (A.15.1).
Transmitting and acquiring develop separately. Under hard labels, same-checkpoint transfer across 15 checkpoints of the 160M models stays weak until a late, sharp rise, although teachers already hold the preference almost perfectly at early checkpoints (§3.3). Across cross-checkpoint grids, a teacher factor times a student factor, plus a proximity term, explains 95–98% of in-sample variation and 90–97% held out, over thirteen 13×13 Pythia grids (§3.3, Table A15). Under soft labels, a 2k-step teacher transmits to later students in all 33 grids, while later teachers transmit almost nothing to a 2k student. The authors place the onset of transmitting between 1k and 2k steps and of acquiring between 2k and 4k (§3.3). Reversing teacher and student often changes how much transfers (A.16).
What carries the signal. Hard labels: synthetic sequences that keep only the teacher's first-number frequencies, each number drawn independently, retain 40–92% of same-model transfer across four animals and three Pythia-160M variants, but 5–36% for elephant in four OLMo-1B models (§3.1, Tables A6–A7). Soft labels: removing the 43 animal-name tokens from a top-10 loss cuts same-model Pythia-160M transfer sharply but leaves it above the control (§3.1). From Pythia-160M into Pythia-410M almost none survives the removal (§3.2).
Other results. With plain SGD, transfer occurs under LoRA and under full fine-tuning; AdamW gives none under full fine-tuning at any of eight learning rates (A.6). Retraining Pythia-160M with the same initialization and data order on different hardware reduced transfer about as much as changing data order or initialization (§3.2, A.4.2).
Why it matters
The shared-base clause. The Cloud et al. finding states that transmission requires a shared base model. It also records that what "shared base model" means was not specified in the summary it was drafted from. This paper measures the parts separately. On small base models with an animal preference, transfer survives a change of initialization, data order, corpus, size and, between similar tokenizers, architecture. The clause holds as a tendency for hard labels: a shared base makes transfer much stronger and far more reliable. It does not hold as a requirement. Among Pythia-160M variants under soft labels it does not hold even as a tendency; across sizes and families a different-model teacher beats the same-model teacher in 13 of 30 pairs.
Beyond Owls' cross-base null. Beyond Owls kept a shared base throughout. Its one cross-base test, letter counting from a Qwen teacher into a Nemotron student, showed no transfer. That result and this paper do not test the same thing. One is a skill, transferred between much larger models from different families. The other is a preference in small base models, found by sweeping hyperparameters. Here cross-model hard-label transfer appears in a minority of a sweep. A single setting could miss it. On this paper's evidence, the Beyond Owls null reads as setting-specific, not as confirmation of the requirement. This is the wiki's reading; neither paper compares the two.
A developmental account the wiki did not have. The concept treats transmission as one mechanism with teacher and student roles. This paper finds that the two roles are separate abilities, with different onsets in pretraining and an approximately multiplicative combination. That is a structural claim about the mechanism, made by holding one role fixed while varying the other across checkpoints.
On-policy distillation as a channel. Filed entries cover sampled tokens and Beyond Owls' logit distillation on teacher-written text. Here the student writes its own text and the teacher only scores it, a setup the authors tie to current post-training practice. That channel gave the most consistent transfer in the paper.
Necessary and sufficient are separate questions. The Cloud et al. entry also records a same-base reproduction by van der Weijden et al., with one model in which transfer was almost absent. That bears on whether a shared base is sufficient. This paper bears on whether it is necessary.
interpretive tensions
Existence against typical size. The hard-label cross-model claims say that some teacher and student settings transfer, more often than chance. They do not say a typical cross-model distillation transfers. The authors report medians near zero across size and architecture. They also note that the strongest cross-model hard-label results often come after extended student training that degrades language modelling to some extent. One condition fails individually: lion, with different initialization and the same data order. The abstract's "transfer persists" covers both kinds of evidence without distinguishing them.
The soft-label channel is less hidden. The training text contains no animal, but the loss compares full-vocabulary distributions. At every position it therefore includes the teacher's probability for the animal's own token. In the top-K variant, removing the 43 animal-name tokens from the loss cuts transfer sharply (it stays above control same-model, almost none survives 160M→410M). An LLM judge also finds slightly more animal references in the students' screened text than in the controls': 0.29% against 0.09%. The authors read this as within a margin they fixed before the audit. The channel whose cross-model results are most consistent is thus the one where the signal is least separate from the trait's own token. Whether that is subliminal learning in Cloud et al.'s sense, or a softer distillation of the preference itself, the paper leaves open, and so does this entry.
What replaces "shared base". Every cross-model pair shares a tokenizer; the soft-label size and family pairs are restricted to the leading block of token IDs on which both tokenizers agree; and the one hard-label architecture test pairs tokenizers the authors call similar. No pair tests a genuinely mismatched vocabulary. The paper cites a toy-MLP result in which compatible output heads, rather than matched hidden-layer initialization alone, can decide whether transfer happens. It does not itself test tokenizer or vocabulary mismatch. The condition may have moved from the weights to the output space. Nothing here establishes that.
Small base models and a token-level preference. The trait is a next-token probability after a first-person prefix, in base models of 160M to 6.9B parameters, most work at 160M. The authors say mechanisms and strength may differ in larger or instruction-tuned models and for other traits. How far a base model's completion preference is a psychological trait at all is the same question the concept's animal-preference instantiations already carry.
Dependence on unreleased, partly retrained models. Code is not yet public. The different-initialization, same-order condition and several asymmetry results use the authors' own retrained Pythia models. They show that hardware alone changes transfer.
concepts
- Subliminal learning — third instantiating finding, and the first designed around the shared-base condition. An animal preference transfers in small base models across initialization, data order, corpus, size and architecture. Under soft labels this is a consistent effect. Under hard labels it is a tail-exceedance result, against a much stronger same-model effect. It also adds two structural claims: on-policy soft-label distillation as a channel, and transmitting and acquiring as separate abilities that develop at different points in pretraining.
cross-references
- Emergent capabilities — not instantiated. The paper's "through pretraining" is about when the abilities to transmit and acquire appear across checkpoints. It is not about pretraining-corpus composition, the sub-shape the open question about this concept's relation to subliminal learning turns on. The entry leaves that question where it is.
sources
- Tang, A., Anand, V., & Paulo, G. (2026). Tracing Subliminal Learning Through Pretraining. arXiv:2610.30884 (v1 22 October 2026, read).
- van der Weijden, D., Brack, N., & Báez Santamaría, S. (2026). Reproducing and Evaluating the Generalizability of Subliminal Learning in Open-Weight Models. arXiv:2609.12586.