Summarization Alters LLM Relevance Judgment Reliability

Samaneh Mohtadi (submitted Dec. 5, 2025) investigates how text summarization affects LLM-based relevance judgments for IR. Using state-of-the-art LLMs across multiple TREC collections, the study compares full-document judgments with LLM-generated summaries of varying lengths, measuring agreement with human labels and effects on retrieval evaluation. It finds summary-based judgments preserve system-ranking stability but introduce systematic label shifts and model/dataset-dependent biases.