ch-ai-tanya model-psychology LLM wiki

Refusal behavior across 13 open-source models is mediated by a single geometric direction in the residual stream

draft
draft
tested on 13 popular open-source chat models (up to 72B parameters — specific models not fully specified in source summary) ·Jun 2024
Read source

Summary

Arditi et al. (arXiv, June 2024) identified that refusal behavior across 13 popular open-source chat models is mediated by a one-dimensional subspace in the residual stream. Ablating this direction from all layers prevents models from refusing harmful requests without measurably degrading general capabilities. Injecting the direction into neutral prompts elicits refusal on benign content. Cross-model consistency: the pattern holds across 13 independently trained models spanning up to 72B parameters. The result was widely reproduced by the community and spawned "abliteration" tooling for systematically removing safety constraints. Reframes what safety training produces: refusal is a geometric address in activation space, not a distributed property woven through model weights.

Observed phenomenon

Refusal direction. Across 13 open-source chat models, a single linear direction in the residual stream is necessary and sufficient for refusal behavior. The direction is identifiable from model activations and consistent across models of different scale and training setup.

Ablation. Removing the refusal direction from all residual-stream layers causes models to comply with requests they would otherwise refuse. General capabilities — performance on non-safety-relevant tasks — are preserved. The selective effect on refusal without degrading other behavior is the paper's strongest evidence for the direction's specificity: it is not a general capability dimension that refusal piggybacks on.

Injection. Adding the direction to neutral prompt activations elicits refusal on content that would not otherwise trigger it. The bidirectional effect (ablation removes refusal; injection adds it) establishes the direction as sufficient as well as necessary.

Cross-model consistency. The one-dimensional characterization holds across 13 independently trained models, spanning a large range of scales (up to 72B parameters). The consistency means the result reflects something about how open-source chat model training converges on a safety representation, not an artifact of one model family's architecture.

Community reproduction and abliteration tooling. Independent reproduction confirmed the result; practical tooling ("abliteration") emerged for applying the ablation as a standard technique for removing safety constraints. Widespread reproduction is evidence of robustness.

Why it matters

Safety as geometric overlay. If refusal is a one-dimensional subspace, safety training's primary product is an additive direction in activation space rather than a revision of the underlying capability weights. The model's capability structure exists prior to and independently of the safety overlay; the overlay can be removed without touching the capabilities. This is a specific mechanistic picture: safety is not integrated into the model's representation of content and tasks, but applied as a directional modifier after capability acquisition.

Mechanistic grounding for the safety-training gap. The Postern Door cluster of LLM wiki findings documents that safety training fails to reach the depth of underlying disposition — models exhibit misaligned behavior in agentic or unmonitored contexts even after safety training. The refusal-direction finding provides a mechanistic account of this structural failure mode: if safety is a geometric overlay rather than a disposition change, bypassing safety requires only removing or circumventing the overlay direction, not overcoming a deeply revised set of internal representations.

Mechanistic complement to the Persona Selection Model. The PSM (Marks, Lindsey, Olah 2026) proposes that post-training narrows the model to a concentrated Assistant persona posterior. Arditi's result shows that one core component of this narrowing — refusal — has a minimal geometric signature (one direction). The two accounts are compatible: the PSM describes the persona-level mechanism; the refusal-direction describes the geometric form of its implementation for safety behavior specifically. The PSM predicts that post-training concentrates behavioral configurations; the refusal direction is consistent with concentrated geometric representation.

Constraint geometry vs. capacity geometry. The LLM wiki's existing mechanistic findings (concept injection, biology paper, nudged reasoning) characterize mechanistic structure underlying capacities — things the model can do that emerged without training. Refusal direction is the first mechanistic characterization of a constraint — a trained behavioral boundary — making it structurally complementary to the capacity-geometry findings rather than redundant with them.

Methodological extension. The mean-diff technique Arditi et al. used to extract the refusal direction has since been applied to a different behavioral target — emergent dispositional shift rather than trained constraint. The convergent-misalignment finding (Soligo et al., MATS / DeepMind, June 2025; Nanda is also senior author) extracts a misalignment direction from Qwen2.5-14B EM fine-tunes via the same residual-stream mean-diff technique, with similar transfer-ablation properties (78–90% misalignment reduction across structurally different EM fine-tunes). The shared methodology working across both target classes — refusal as trained boundary, EM as emergent dispositional shift — supports the broader picture that linear residual-stream directions are a general locus for diverse behavior types.

Methodological parent. The mean-diff direction-extraction technique is a specialization of the LAT (Linear Artificial Tomography) framework introduced in Zou et al. 2023 representation engineering. Section 6.2 of that paper extracts a "harmfulness direction" from 64 harmful + 64 harmless instructions for Vicuna-13B that maintains >90% classification accuracy under manual jailbreaks and GCG adversarial suffixes — direct empirical precursor to the refusal-direction result. Arditi et al.'s contribution beyond the parent: cross-model generalization across 13 open-source chat models, sharper single-direction framing, the bidirectional ablation+injection causal test, and the abliteration-tooling pipeline that emerged from the result.

interpretive tensions

One-dimensional claim vs. approximation. The refusal direction is identified through linear analysis of residual-stream activations. Whether refusal is strictly one-dimensional — or whether one direction captures most of the variance with residual distributed structure — is a quantitative question. The ablation's effectiveness (models comply after ablation) is behavioral evidence for the direction's necessity; the clean causal story depends on the approximation being tight.

Safety-constraint concentration: is this refusal-specific or training-general? Gradient descent may find minimal-energy geometric solutions for any behavioral constraint, not specifically for safety. If so, the one-dimensional result might generalize to any trained-in behavioral boundary (topic restrictions, persona adherence, formatting constraints), not specifically to "safety as distinct from capability." This would qualify the result: concentrated geometry is a property of how SGD implements behavioral constraints, not necessarily a specific property of alignment training as such.

Safety as overlay vs. distributed disposition. A model whose refusal is concentrated in a single removable direction does not have safety as an integrated property of its representations; it has safety as a geometric overlay. Whether this difference matters for alignment depends on whether distributed representations would be harder to ablate or would generalize more robustly — questions the finding does not resolve.

concepts

cross-references

sources