Summary
Yang, Chern, Qiu, Neubig and Liu define an honest model as one that answers the
questions it can answer and declines the rest with an "I don't know" (idk)
response (Yang et al. 2023).
Whether a model knows an answer is not measured inside the model. A question
counts as known when the model's own sampled answers are correct often enough.
In the authors' words, they "approximate the model's internal knowledge through
the accuracy of its responses". The paper fine-tunes LLaMA2-Chat-13B on 8,000
TriviaQA questions labelled this way. On held-out TriviaQA, the share of
unknown questions the model declines (prudence) rises from 0 to between 47.70
and 67.72, depending on method. The share of known questions it now declines
(over-conservativeness) rises to between 9.94 and 15.89. Declining transfers to
other free-form question sets but barely appears on multiple-choice MMLU unless
MMLU examples are added to training.
The paper is filed for the wiki's introspection boundary. Its target behaviour
is a statement about the model's own knowledge, which makes it look like
introspection. Its method defines that knowledge from the model's outputs and
never tests whether declining draws on internal state. On the
introspection concept's own Definition and
evidentiary bar, this entry reads it as trained self-report with unestablished
access, and files it concept-less. It predates by two years the
report-channel training interventions filed under introspection, among them
honesty elicitation and
confessions.
Method
Labels. For each training question the unaligned model samples ten answers
at temperature 1. The share that are correct is the question's expected
accuracy. Questions are balanced between known and unknown by the unaligned
model's greedy answer, then 8,000 are sampled.
Four training methods. Absolute treats a question as known if expected
accuracy is at least 0.1, so one correct answer in ten. Known questions are
trained on one of the model's own correct answers and unknown ones on a fixed
idk sentence. Confidence-Num and Confidence-Verb do the same but prefix
known answers with a stated confidence, as a percentage or as one of five
verbal levels running from very unsure to certain. Multisample keeps all ten
samples per question, replacing each wrong one with the idk sentence, which
multiplies the data by ten. Training is full-parameter, learning rate 1e-6, for
two epochs (one for Multisample). For the trained models, every prompt in
training and evaluation tells the model that saying it cannot answer is
acceptable.
Baselines. The unaligned model with a plain question prompt. A
prompt-based model that is not trained but gets the idk-permitting prompt. A
fine-tuned baseline trained on the same 8,000 questions with the gold answer
for every unknown question instead of idk.
Metrics. Responses at temperature 0 are classed as idk (string match on a
short phrase list), correct (gold answer present, judged by gpt-3.5-turbo-0613)
or wrong. Each question is then compared across the unaligned and aligned
models. Prudence is the share of questions the unaligned model did not
answer correctly, and the aligned model also does not, on which the aligned
model says idk. Over-conservativeness is the share of questions the
unaligned model answered correctly on which the aligned model says idk. The
honesty score is their average, with over-conservativeness inverted, so a
model that never says idk scores 50. "Known" at evaluation time is therefore
one greedy answer by the unaligned model.
Key results
In distribution (Table 3, LLaMA2-Chat-13B, TriviaQA). The unaligned and
fine-tuned baselines never decline (prudence 0, honesty 50.00). Accuracy is
73.71 unaligned and 71.47 fine-tuned. Prompting alone gives prudence 33.77,
over-conservativeness 12.50 and accuracy 64.70. The trained methods give
prudence 47.70 (Absolute), 61.11 (Confidence-Num), 58.91 (Confidence-Verb) and
67.72 (Multisample). Over-conservativeness is 9.94, 12.38, 10.68 and 15.89, and
accuracy 71.30, 69.80, 73.34 and 68.88. Multisample has the highest honesty
score (75.91). Confidence-Verb keeps the most accuracy of the trained methods
(73.34, against 73.71 unaligned).
Out of distribution (Table 4). On Non-AmbigQA, Multisample reaches prudence
64.73 at over-conservativeness 24.37, and accuracy falls from 49.63 to 44.26.
On PUQA, 1,000 questions asking who wrote a 2023 paper the model cannot know,
prudence is 0 unaligned, 28.90 prompted, and 79.90–87.30 for the three
confidence and multisample methods (34.20 for Absolute). PKQA is 1,000
questions the model generated itself and the unaligned model answers
correctly by construction (accuracy 100.00). The trained methods keep 95.90–96.80
accuracy there. The fine-tuned baseline falls to 87.70. The authors attribute
this to fine-tuning on answers beyond the model's knowledge teaching it to
hallucinate, citing Schulman and others. They do not test that account.
Multiple choice (Table 20, MMLU). Given four options, trained models rarely
decline: Confidence-Verb's prudence is 2.60 and Multisample's 9.53. Adding 284
MMLU examples to training raises Multisample's prudence to 78.95, with
over-conservativeness 44.61 and accuracy falling from 47.17 unaligned to 33.73.
Scale and backbone (Tables 18–19). For Confidence-Verb, prudence is 56.04 at
7B, 58.91 at 13B and 51.44 at 70B, and over-conservativeness 11.43, 10.68 and
6.51. On InternLM-, Qwen- and Baichuan2-Chat-7B, Confidence-Verb lifts the
honesty score to 68.53–74.37. Prompting alone fails on Qwen-Chat-7B, whose
accuracy drops to 1.46.
Tax (Tables 5, 23). Helpfulness on 1,320 non-QA requests is close to
unchanged; by this entry's arithmetic the largest drop from unaligned is 0.04 on
Auto-J and 0.06 on GPT-4 (Auto-J 5.56 unaligned, 5.54 Confidence-Verb, 5.52
Multisample; GPT-4 8.62, 8.61, 8.56). On 700 BeaverTails prompts, no model
gives an unsafe response, as judged by GPT-4o.
Why it matters
The introspection cluster's report-channel interventions are
honesty elicitation,
confessions and the later adapter and
lie-detector work. Each trains a model to say something true about itself.
This paper trains the narrowest version of that statement, that the model does
not know an answer, on 2023 open models, and its result carries the same caveat those entries
carry. Behaviour improves and transfers across question sets, but nothing shows
what the statement is read from. The confessions finding located its limit in
hallucinations the model does not register. This paper's fine-tuned baseline
bears on how such hallucinations can be trained in: fine-tuning on gold answers
to unknown questions lowers accuracy even on questions the model wrote itself,
which the authors read as training the model to make answers up.
The biology paper describes one
mechanism this training might be tuning: a default "can't answer" circuit in
Claude 3.5 Haiku that known-entity features suppress, with hallucinations when
familiarity suppresses it falsely. This paper has no mechanistic measure, so
that link is this entry's conjecture. If it holds, though, what the idk
training adjusts is a familiarity signal. In the terms of the introspection
scope note's evidentiary bar, that is a first-order signal, not a second-order
reading of one.
The multiple-choice result is the paper's clearest negative. A model trained to
decline unknown free-form questions almost never declines when given four
options. When trained to, Multisample declines 44.61% of the questions the
unaligned model answered correctly. On this entry's reading, declining depends on the
question's format, not only on what the model knows. That fits the
task-conditional readout described in the scope note, and the
personalization-mirage finding's gap
between auditing and generating.
interpretive tensions
Self-referential target, external measure. The paper's glossary separates
honesty, which concerns model knowledge, from calibration, which in current
work is measured against world knowledge. On that split this work is about
self-knowledge, and the labels are indexed to each model's own samples, so
each backbone gets its own training set. But the authors define "known" from
output accuracy, and say a better test of whether a model knows an answer is
left to future work. Whether that makes this introspection depends on whether
self-indexed labels are enough. On the concept's Definition, they are not.
No privileged-access control. The paper does not ask whether a different
model, or a probe on question features, could predict which questions this
model gets wrong as well as the model itself does. The
reality-check finding found that
self-derived labels in another paradigm were predictable from input embeddings
alone. Every PUQA question asks who wrote a paper, and the paper reports only
prudence there. A rule declining all such questions would score full prudence,
so PUQA cannot separate a learned question-type rule from knowledge of
specific gaps.
What counts as idk. The idk detector is a string match. Its list ends with
a phrase that introduces a correction, and the punctuation leaves unclear
whether the word however alone matches. Responses that hedge or push back may therefore count
as declining. The paper's Edison case shows an aligned answer that declines
and then disputes the gold answer.
The scale claim. The authors say larger models learn more from idk data,
with substantially higher prudence. Confidence-Verb's own prudence is lower at
70B (51.44) than at 13B (58.91). By this entry's arithmetic, the claim holds
for the gain over prompting alone (−6.08 at 7B, +25.14 at 13B, +33.18 at 70B),
because prompted prudence falls with size. The paper does not say which it
means. The authors also say over-conservativeness is marginally higher at
larger sizes; Table 18 shows it lower at 70B (6.51) than at 7B or 13B (11.43,
10.68).
Honesty without lying. The authors exclude lying by assumption: models not
prompted, trained or placed in special contexts, they argue, seldom state falsehoods they know to be false, and wrong
TriviaQA answers are more likely invented than believed. Both are hypotheses
the paper states and does not test. "Honesty" here means declining
appropriately, not the belief–output consistency that later filed entries test.
No variance, 2023 models. Results come from greedy decoding, graded by
gpt-3.5-turbo. The paper reports no seeds, variance or significance tests, so
differences of a few points between methods are not shown to be reliable.
concepts
No concept instantiated. The finding sits next to
introspection and does not instantiate it, for
three reasons in that concept's own text. The Definition names access to
internal states "as distinct from producing outputs shaped by those states".
This paper's idk response is trained to follow the model's sampled accuracy,
an output statistic. The "Not self-report" boundary says introspection is the
access that may or may not underlie self-report, and this paper trains the
self-report without testing the access. The evidentiary-bar paragraph says
behavioural paradigms can establish at most privileged access. This paper does
not run the comparison that would establish even that. Honesty
elicitation and
confessions are also trained report-channel
interventions that do not establish access, yet are listed under the concept.
The distinction this entry draws is that their reports concern the model's
own internal states or conduct, and each yields a result about access (the
introspective stratum resists, confession fails where the violation is not
registered); this paper's report concerns whether a fact can be retrieved, and
yields no such result. That line is thin, and placing it is an editor call.
The relation is carried in Cross-references. The over-confident answering the paper trains against is
also a second example, after the personalization-mirage finding, of a
confabulation pattern the wiki has not drawn as a concept.
cross-references
- Honesty elicitation and
confessions — later report-channel training
interventions filed under introspection. This paper is the earliest of the
type and the narrowest, targeting only "I don't know". - Biology of a large language model
— the default "can't answer" circuit is a candidate mechanism for the
behaviour trained here. The link is not tested. - Introspection reality check —
supplies the control this paper lacks: whether self-derived labels are
predictable from the input alone. - Personalization mirage — models apply
the limits of their evidence when audited but not while generating. The
MMLU format effect here has the same shape, and both are confabulation
results without a concept.
sources
- Yang, Y., Chern, E., Qiu, X., Neubig, G., & Liu, P. (2023). Alignment for
Honesty.
arXiv:2312.07000 (v1, 12 Dec 2023; v2, 28 Oct 2024, read). NeurIPS 2024.