RLMF trains language models to express uncertainty more faithfully
A Yale University and Google Research paper accepted at COLM 2026 introduces reinforcement learning with metacognitive feedback (RLMF), a post-training method that gives more weight to completions when a model accurately judges its own performance. Across Qwen3 and Llama 3.1 models and 10 tasks, the authors report better faithful-calibration scores than standard reinforcement learning and prior baselines while preserving task accuracy in their evaluations. The method also uses self-assessment to select training examples and turns numerical confidence into natural-language uncertainty. The results are promising but preliminary: calibrated uncertainty is not factual verification or broad machine self-awareness.
What the researchers changed
A June 30 paper from Yale University and Google Research introduces reinforcement learning with metacognitive feedback (RLMF), a post-training method designed to make a language model's expressed uncertainty better reflect its estimated internal confidence. The work is listed among the accepted papers for COLM 2026.
RLMF modifies the learning signal used during group relative policy optimization. Candidate completions still receive rewards for task performance and faithful confidence. The method adds another signal: how accurately the model predicts its own performance. Among completions that already perform well on the main objective, those with better self-assessment receive a stronger advantage during training.
The researchers pair that mechanism with metacognitive data selection. The model estimates how well its internal and expressed confidence align, and training examples are selected from both the high- and low-scoring ends. A separate editing stage then converts sentence-level numerical confidence into more natural, context-sensitive language.
Results reported by the authors
The team evaluated Qwen3 models at 1.7B, 4B and 8B parameters and Llama 3.1 8B Instruct across 10 tasks. Training used one dataset at a time, while evaluation covered question answering, reasoning, hallucination detection and answerability tasks.
| Model setting | Average faithful-calibration score | Accuracy |
|---|---|---|
| Llama 3.1 8B baseline | 0.60 | 0.31 |
| Llama 3.1 8B + standard RL | 0.77 | 0.40 |
| Llama 3.1 8B + RLMF | 0.84 | 0.41 |
| Qwen3 8B baseline | 0.54 | 0.55 |
| Qwen3 8B + standard RL | 0.51 | 0.59 |
| Qwen3 8B + RLMF | 0.83 | 0.57 |
The paper reports that RLMF improved its cMFG* faithful-calibration metric by as much as 63% over standard RL in a tested setting. That is a best-case calibration result, not a 63% accuracy gain or hallucination reduction. Its metacognitive data-selection strategy also outperformed random selection and an active-learning-style baseline on the same metric. In a 120-example expert evaluation, the authors report 95% to 98% win rates over a prior fine-tuning baseline across four qualities of uncertainty wording.
LDS assessment
The practical contribution is not that a model suddenly knows the truth. It is a training recipe for reducing the mismatch between how certain a model sounds and how stable its answers appear under repeated sampling. A model can communicate uncertainty more faithfully and still be wrong, biased or unsupported.
The evidence is promising but not yet production proof. The main experiments cover two open-model families up to 8B parameters; intrinsic confidence is estimated from repeated samples and model-based judgments; and cMFG* is a new metric introduced by the same research team. The pipeline also requires multiple generation, judging, training and rewriting stages, so teams should measure compute cost before adopting it.
For practitioners, RLMF is best viewed as a post-training and calibration technique, not a replacement for retrieval, external verification, safety evaluation or human review. The strongest next evidence would be independent reproduction on larger models, adversarial testing for reward hacking and excessive hedging, and deployment studies showing that people make better decisions when models express uncertainty this way.
Key Points
- 1RLMF scales the reinforcement-learning signal using the accuracy of a model's self-assessment after a completion clears the main task objective.
- 2The authors report sizable faithful-calibration gains across 10 tasks and two open-model families, and released the implementation publicly.
- 3Faithful uncertainty is not factual correctness; independent replication and real-world deployment evidence are still needed.
Scoring Rationale
The COLM-accepted work proposes a technically novel post-training signal, reports broad calibration gains across 10 tasks, and releases code. Its practical impact is moderated because the main evidence comes from the authors' experiments on models up to 8B parameters and needs independent production-scale replication.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
