Researchers Introduce Thermodynamic Faithfulness Metrics For LLMs
A Dec. 4, 2025 preprint by Igor Halperin proposes two unsupervised metrics—Semantic Faithfulness (SF) and Semantic Entropy Production (SEP)—to evaluate LLM faithfulness. The method models question–context–answer triplets as topic distributions and infers transition matrices using KL divergence and convex optimization; SEP measures thermodynamic entropy production. The authors demonstrate the framework on LLM summarization of corporate SEC 10-K filings.
Key Points
- 1Introduce two unsupervised metrics, Semantic Faithfulness (SF) and Semantic Entropy Production (SEP), for LLM faithfulness evaluation.
- 2Model Q-C-A triplets as topic distributions and measure KL divergence between inferred transition matrices to quantify faithfulness.
- 3Enable unsupervised hallucination detection and control, demonstrated on LLM summarization of SEC 10-K filings.
Scoring Rationale
Novel, broadly applicable unsupervised faithfulness metrics with practical demonstrations; limited by single preprint submission and pending peer review.
Sources
Public references used for this report.
Practice with real Logistics & Shipping data
90 SQL & Python problems · 15 industry datasets
250 free problems · No credit card
See all Logistics & Shipping problems
