Researchllmevaluationframing biasprompting
LLMs Exhibit Framing Bias In Evaluations
8.0
A January 20, 2026 arXiv preprint by Yerin Hwang et al. investigates framing bias in LLM-based evaluation, testing symmetric predicate-positive and predicate-negative prompts across four high-stakes tasks. The study measures responses from 14 LLM judges and finds significant, systematic discrepancies with model families showing distinct agreement or rejection tendencies. The authors conclude framing is a structural bias, recommending framing-aware evaluation protocols.
Key Points
- 1Demonstrates framing manipulates LLM judgments across four high-stakes evaluation tasks using symmetric prompts.
- 2Finds 14 LLM judges show systematic susceptibility, with model families tending toward agreement or rejection.
- 3Impacts evaluation reliability, suggesting need for framing-aware protocols and standardized prompt designs.
Scoring Rationale
High novelty and broad scope drive score, limited by preprint status and modest direct mitigations.
Sources
Public references used for this report.
Practice with real Logistics & Shipping data
90 SQL & Python problems · 15 industry datasets
Used by DS/ML engineers at top companies
High-Value Overnight OrdersEasyDelivered International ShipmentsMediumOn-Time Delivery Rate by CarrierHard
250 free problems · No credit card
See all Logistics & Shipping problems
