Topic-Aware Models Summarize Lived Healthcare Stories

An arXiv preprint submitted 23 Oct 2025, titled "Topic-aware Large Language Models for Summarizing the Lived Healthcare Experiences Described in Health Stories," evaluates a pipeline that combines topic modeling and hierarchical LLM summarization on qualitative narratives (arXiv:2510.24765). Per the paper, the authors applied Latent Dirichlet Allocation (LDA) to 50 transcribed stories from African American storytellers and identified 26 topics. The study produced story-level summaries and then aggregated those into topic summaries using an open-source LLM-based hierarchical summarization approach, as described in the preprint. The paper reports that GPT-4 rated those topic summaries as free from fabrication and as highly accurate, comprehensive, and useful, and states that GPT-4 ratings showed moderate-to-high agreement with assessments from two domain experts.

Per the arXiv preprint, topic discovery used LDA to cluster narrative content into topical groups, after which the authors generated story summaries and then topic summaries by summarizing across the story summaries. The evaluation combined automated assessment using GPT-4 with expert validation by two domain experts, and the authors list topics including health behaviors, interactions with medical team members, caregiving, and symptom management (arXiv:2510.24765). Editorial analysis - technical context: Combining unsupervised topic models with LLM summarization is a known pattern for structuring heterogeneous qualitative datasets, because topics provide an intermediate abstraction that can focus LLM outputs and support finer-grained evaluation.

For practitioners working with patient narratives and other unstructured health data, the paper demonstrates a repeatable pipeline for extracting topic-level insights from a small corpus of stories. The study's use of both GPT-4 for automated quality checks and independent expert review illustrates a pragmatic hybrid evaluation approach that other teams can adapt when labeled evaluation data are scarce. Observed patterns in similar work suggest that topic-aware summarization can improve interpretability and help prioritize qualitative themes for downstream analysis or hypothesis generation.

Follow-up work to check generalization beyond the small, demographically specific dataset used here, replication on larger and more diverse story corpora, and open releases of the summarization prompts, prompts templates, or code. Also watch for evaluations that replace or augment GPT-4 ratings with quantitative human annotation protocols to measure reliability across raters and settings.