Anthropic published a randomized controlled trial in 2024 finding developers using AI coding assistance scored 17% lower on comprehension quizzes than those coding manually, with productivity gains failing to reach statistical significance. The 52 mostly junior engineers learned the Trio asynchronous library; patterns of AI use determined outcomes, with conceptual-question usage linked to 65%+ scores while full code delegation yielded under 40%, and a University of Maribor study found similar correlations.
Key Points
- 1Reports show AI-assisted developers scored 17% lower on comprehension quizzes in RCT of 52 engineers
- 2Highlights that AI code generation reduces conceptual learning, especially debugging skills, despite minor time savings
- 3Recommend using AI for explanations and guided tutoring, not full code delegation, to preserve learning
Scoring Rationale
Strong experimental evidence and practical guidance, but small sample size and short-term measurement limit generalizability.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems