Anthropic urges government power to block risky AI models
In a policy paper titled "Policy on the AI Exponential," Anthropic argues that governments should have legal authority to block or deter deployment of AI models that could cause catastrophic harms, including biothreats and major cybersecurity vulnerabilities, per the document. Anthropic proposes civil penalties tied to global annual revenue that escalate for repeated violations and recommends frontier models be subject to testing, public transparency of findings, independent evaluation, and robust security programs, according to the paper. The proposal also sets quantitative thresholds, models trained with more than 10^25 FLOPs or developed by firms with more than $500M in AI revenue or more than $1B in AI R&D spending, for applying the rules, per Anthropic's published policy.
What happened
Anthropic CEO Dario Amodei published a two-part policy paper, "Policy on the AI Exponential," on June 10, 2026. Per the paper, "When a model poses risks of this kind, the government should have the legal authority to block or deter its deployment - beyond what exists in current law or in existing proposals in Congress - with civil penalties tied to global annual revenue that escalate with repeated violations." The paper covers two frameworks: an Advanced AI Framework addressing catastrophic-risk governance, and an Economic Policy Framework addressing worker displacement and sharing the financial gains of AI.
Technical scope
Per Anthropic's proposal, the regulatory regime applies to models trained using more than 10^25 floating-point operations (FLOPs), developed by companies earning more than $500M in AI-related revenue or spending more than $1B on AI R&D. The paper identifies four categories of catastrophic risk: biological risk, cyber risk, loss-of-control risk, and automated R&D. Frontier developers would be required to test models, publish findings, submit to independent evaluation, and maintain security programs.
Why now
Anthropic cites Claude Mythos Preview's confirmed discovery of thousands of high-severity vulnerabilities across every major operating system and browser as evidence the pace of capability advancement now justifies binding policy rather than transparency alone. The paper argues existing state transparency laws are not sufficient and calls for federal authority with enforcement teeth.
Economic framework
Beyond safety, Amodei proposes wage insurance, retention tax incentives, expanded social safety nets, and possible universal capital accounts to address AI-driven labor displacement, according to Axios reporting. He frames opposition to data center buildouts as "largely a symbol or outlet for broader economic anxieties about AI."
Editorial context
Industry observers note that compute- and revenue-based thresholds aim to concentrate oversight on a small number of frontier developers, but create auditability and gaming risks. The proposal is also sure to attract criticism that Anthropic is proposing rules that lock in its own dominance. Practitioners should watch for Congressional response, standardization of FLOPs measurement, growth of certified third-party evaluation labs, and international alignment on comparable frameworks.
Key Points
- 1Anthropic's paper calls for government authority to block high-risk models, framing a regulatory approach tied to capability and company-size thresholds.
- 2Threshold-based rules (compute, revenue, R&D spend) aim to limit scope to frontier systems but create measurement and auditability challenges for practitioners.
- 3If adopted, the proposal would increase demand for independent evaluation, secure development practices, and reproducible testing infrastructure across the AI ecosystem.
Scoring Rationale
Major policy statement directly from Anthropic's CEO with specific regulatory architecture - compute/revenue thresholds, blocking authority, enforcement mechanisms - published alongside Claude Fable 5 and Mythos releases and covered by Bloomberg and Axios. Significant for practitioners building evaluation and compliance infrastructure, though it is a proposal rather than enacted law.
Sources
Public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
