Summary
Sun, Zhang and Sheng measure over-inference — personalized models
fabricating user attributes beyond the evidence — across 12 models, 7 families
and 143,616 judged claims. Every model does it, on 35–49% of claims. The
finding that matters for this wiki is not that rate but a gap beside it: asked
directly to list the inferences it has drawn and link each to evidence, the
same model produces unsupported claims at 0.7–4.6%. A mean gap of 38.6 points
separates what a model asserts while personalizing from what it will endorse
when asked to audit. As the authors put it, the same knowledge is not equally
applied across the two settings.
This is an independent behavioural test of the reading
introspection adopted one day earlier on
mechanistic grounds. The global-workspace finding
argued that access is task-conditional — the same information enters or bypasses
the reportable subspace depending on what the model is asked to do with it —
and prompted a scope-note revision saying so. This paper reaches the same shape
from the outside, with no interpretability apparatus, a different institution,
and a task the wiki had not looked at: the knowledge that an inference is
unsupported is available under the audit framing and not applied under the
generation framing. Two findings converging on a claim from unrelated
directions is worth more than either.
Method
MirageBench holds 150 personas balanced across stereotypical,
counter-stereotypical and neutral profiles, and six personalization tasks
arranged along what the authors call an imagination gradient. Claims are scored
by an independent judge into a four-way faithfulness taxonomy — grounded,
reasonable, stereotyped, fabricated — with over-inference the union of the
latter two. Judge validity is checked against a blind human annotator on 400
claims: Cohen's κ=0.863 four-class, κ=0.900 binary.
Three elicitation conditions do the work. Task generation is the model
personalizing normally. Task self-audit asks the model to score its own
generated claims. Probe asks it directly to list its inferences with
evidence links. The same judge scores all three, so the gap between conditions
is not a measurement artifact of differing graders.
Key results
Over-inference is universal in this panel. All 12 models over-infer on
35–49% of claims; cross-model mean 41.6%, claim-weighted 41.8%. Rates are
task-dependent, 27% to 59%, highest on apartment-and-home descriptions.
The generation/audit gap is large. Under Probe, incorrect rates fall to
0.7–4.6% — a mean 38.6-point gap against Task generation. The model can
identify the boundary of its evidence when that is the question; it does not
apply the boundary when the question is to personalize.
The Self-Monitoring Inversion is real but exploratory, and the authors say
so. Across models, self-assessed over-inference is negatively rank-correlated
with judge-measured over-inference: ρ=−0.60, p=0.044 by permutation. Qwen3-8B
self-reports the lowest rate (13.0%) and receives the highest judge rate
(48.7%); Kimi-K2.5 self-reports the highest (58.2%) and lands mid-pack (43.1%).
But the bootstrap CI crosses zero — [−0.90, +0.06] naive, [−0.87, +0.14]
family-clustered — on n=12 with correlated families, and the authors label it
"an exploratory cross-model inversion, not a precisely estimated coefficient".
Within a model, self-audit works. Per-model AUROC for identifying
above-median-judge-OI records runs 0.58–0.83, with 9 of 12 above 0.75. The
inversion is a claim about comparing models, not about whether a model can rank
its own claims. The authors' proposed mechanism is calibration asymmetry:
strict self-labelers (Claude, GLM, Kimi) flag more and also generate more
carefully; lenient ones (GPT-4o-mini, Qwen3-8B) label nearly everything
reasonable and generate less inhibited.
Inferred attributes accumulate. In an 8-round, 2-persona pilot, frontier
models build profiles of 120+ attributes from 3 initial facts, roughly
linearly, with little revision. The authors flag the persona count and a memory
prompt that instructs retention over pruning, and say the usable signal is the
cross-model contrast, not the growth.
Why it matters
The introspection cluster has measured the report channel by asking models
about internal states — injected concepts, their own lies, their own
activations. This finding measures it by asking a model about its own just-
produced claims, which needs no interpretability apparatus and no
privileged access, and finds the same split the mechanistic work found: the
knowledge is there, and whether it is applied depends on what the model is
doing at the time. The scope-note revision saying access is task-conditional
was written from one paper's lens results; this is a second line of evidence
for it, and the kind the revision explicitly asked for.
The within-model / cross-model split is a methodological caution the cluster
should carry. Four report-channel interventions are filed, and their headline
numbers are within-model fidelity measures. This paper shows a case where
within-model self-audit is genuinely informative (AUROC up to 0.83) while the
same quantity compared across models points the wrong way. Any future reading
that ranks models by self-report fidelity is exposed to that inversion — which
is a reason to be careful with the
lie-detector finding's observation that
third-person monitoring beat self-report at every scale tested, since that too
is a cross-model comparison of self-report.
The over-inference phenomenon itself instantiates no concept here. It is a
faithfulness failure with no concealment and no goal — the model is not hiding
anything and gains nothing — so it sits outside scheming and outside
sycophancy, whose shape requires tracking the user's approval rather than
inventing the user. If a confabulation concept is ever drawn, this is a
candidate anchor.
interpretive tensions
The headline result is weaker than its name. "Self-Monitoring Inversion" is
presented as the paper's most consequential finding and is carried in the
title, but its confidence interval crosses zero on both bootstrap variants. The
authors are scrupulous about this in the body. A reader taking the title and
abstract alone would not learn it, and this entry files the inversion as
suggestive rather than established.
Strictness may explain the inversion without any introspective content. The
proposed mechanism — some families label their own inferences harshly, others
leniently — would produce the observed correlation even if no model had any
insight into its own generation. On that reading the inversion is a fact about
label calibration, and the genuinely introspective result is the within-model
AUROC instead. The authors offer the strictness account themselves and do not
separate the two.
The generation/audit gap has a mundane competitor. The Probe condition asks
for inferences with evidence links attached, which is a different and easier
task than generating personalized prose — the framing supplies the standard
along with the question. That the model meets the standard when handed it is
consistent with task-conditional access, and equally consistent with the
standard simply not being salient during generation. Distinguishing these would
need a generation condition carrying the same evidential instruction, which the
paper does not run.
The accumulation pilot is two personas. The authors caveat it twice and
scope the claim to cross-model contrast. It is reported here for completeness
and should not be cited for a rate.
concepts
- introspection — behavioural instantiation,
and the first evidence for the concept's task-conditional reading that does
not come from an interpretability method. Supplies a second, independent line
for the scope note's March-2026 revision, and a methodological caution the
cluster lacked: within-model self-report fidelity and cross-model comparison
of the same quantity can point in opposite directions.
cross-references
- The J-space characterization — the
mechanistic counterpart. Same shape of claim, opposite method: that paper
shows a latent variable causally engaged for report and flexible inference
but bypassed for automatic processing; this one shows knowledge endorsed
under audit and not applied under generation. - Candidate anchor for a confabulation concept. Over-inference is a
faithfulness failure without concealment or goal, which none of the wiki's
current concepts covers. One example, so no proposal.
sources
Sun, Zhang, Sheng 2026,
"The Personalization Mirage: How LLMs Fabricate User Profiles, and Why
Self-Monitoring Misleads", arXiv:2608.04570, 5 August 2026.