Anthropic Runs Silent A/B Tests Degrading Workflow
A paying user discovered that Anthropic is running unannounced A/B tests in Claude Code that modify plan-mode outputs, assigning variants such as "cap" which restricts plans to 40 lines and removes context and prose. The discovery, made by decompiling the binary, shows telemetry logging variant assignments and plan metrics; it implies paying customers are being experimented on without opt-in, disrupting professional workflows.
Key Points
- 1Discovered 'tengu_pewter_ledger' A/B test controlling Claude Code plan-mode output variants, including 'cap'.
- 2Alters planning output by removing context and prose, capping plans at 40 lines, reducing human steering.
- 3Indicates paying users are enrolled without notification, undermining workflow reliability, transparency, and trust.
Scoring Rationale
Reveals significant silent experimentation affecting paid users, providing clear technical evidence, but depends on single-source reverse-engineering.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems