OpenAI Engineers Trace 18-Year-Old Libunwind Race Bug

OpenAI's engineering team spent months chasing intermittent crashes in Rockset, the real-time data system OpenAI acquired in 2024 that indexes ChatGPT's data plugins and conversation search. The crashes looked like a C++ function returning to a corrupted or NULL address, a failure mode rare enough that engineers initially assumed a single root cause and could not reproduce it under controlled testing.

Instead of deep-diving individual core dumps case by case, the team built an automated pipeline, partly written by ChatGPT, that downloaded, parsed, and classified every Rockset core dump generated over the prior year. Population-level analysis split what looked like one bug into two distinct clusters: crashes tied to a single physical Azure host with silent hardware corruption, and a separate, more numerous set tied to C++ exception unwinding. The second cluster traced to an 18-year-old race condition in GNU libunwind, an open-source library used to unwind the stack during C++ exception handling: a single assembly instruction updates the stack pointer before the destination instruction pointer is read, and if a signal arrives in that roughly 100-picosecond window, the kernel can overwrite the in-flight context and corrupt the restored instruction pointer. OpenAI's signal-heavy CPU-accounting mechanism, combined with a recent change that made its signal handler use more stack space, pushed a previously dormant bug into visibility.

The episode is a concrete illustration of a broader debugging principle for infrastructure teams operating at scale: case-by-case root-causing can fail even with strong engineers, while automated, population-level data pipelines can surface patterns a single incident cannot. Because GNU libunwind is used well beyond OpenAI, other organizations running high exception-throughput, signal-heavy C++ services may want to check their unwinder version and dependency chain.

OpenAI switched Rockset from GNU libunwind to libgcc's unwinder as an immediate mitigation, and engineer Nathan Bronson upstreamed a reproducer and fix directly to the GNU libunwind project on GitHub. Watch for whether other large C++ codebases that combine frequent signals with exception-based control flow report similar corruption once the fix propagates into distributions.