Researchers Map Structural Failures in LLM-Generated Code

Researchers formalized what they call the patchwork problem: AI-generated code can look correct in the edited file yet conflict with the wider repository. The paper focuses on structural coherence rather than ordinary logic bugs. A generated route may omit an established guard, a configuration key may never be declared, or one module may consume a field that another module does not produce.

The authors define eight structural failure categories covering imports, internal APIs, dependencies, configuration, resources, control flow, cross-file contracts, and security wiring. Their framework builds multiple repository graphs and applies mature static analyzers where established language semantics matter. It adds purpose-built detectors for cross-file and cross-graph constraints that common tools do not explicitly model.

They evaluated 336 generated changes from two frontier models under four prompting strategies. The paper reports 67 structural failures, with 65 missed by type checking, tests, static application security testing, and regex baselines. The controlled study spans Python and TypeScript repositories and varies how much repository context the model receives. The authors also report a separate validation pass on public AI-generated repositories. These are results from the research team and have not yet been independently reproduced.

Coding agents increasingly modify several files and layers in one task. That makes local correctness an incomplete release criterion. More context can help a model understand the repository, but it does not itself prove that every generated assumption matches the deployed system. A reliable gate needs to inspect the final repository state rather than only the model's explanation or patch diff.

LDS recommends treating repository coherence as a release invariant, with cross-file graph checks and deployment-contract tests added to existing CI. Teams can begin with high-cost failures: undeclared environment variables, nonexistent dependencies, schema producer-consumer mismatches, missing migrations or assets, and routes that differ from their siblings' authorization pattern. Findings should include exact files, lines, and violated contracts so developers can review evidence instead of receiving another opaque score.

The framework is promising because it targets explicit invariants, but independent precision audits and broader language coverage are still needed. The practical lesson is narrower and immediately useful: passing compilation and local tests should not be treated as proof that an agent-generated change is production-ready.