California Discloses Use of Six High-Risk AI Systems

According to the California Department of Technology's annual report, state agencies are currently using six automated decision systems that meet the state's statutory definition of "high-risk" (California Department of Technology report; CalMatters). A year earlier, agencies collectively reported zero high-risk automated decision systems to the state, CalMatters reports. The report lists systems used to predict whether incarcerated people will reoffend, evaluate unemployment-fraud claims, remotely administer exams for California State University students, and detect when students use generative AI in assignments (California Department of Technology report; CalMatters).

The state law the report implements requires agencies to disclose systems "used to assist or replace human discretionary decisions that have a legal or similarly significant effect," including decisions affecting housing, education, employment, health care, and criminal justice (CalMatters citing the statute). The technology department's report also notes that several systems named have been in use for multiple years; CalMatters cites COMPAS as an example of a risk-assessment tool used by the California Department of Corrections and Rehabilitation for over a decade. The report disclosed an additional six systems that were initially flagged as high-risk but were later determined not to meet the statutory threshold (California Department of Technology report; CalMatters).

Public-sector inventories of automated decision tools frequently undercount systems on first pass, particularly when agencies use legacy procurement channels or vendor-classified features. The California Department of Technology says it found more systems this year because it evaluated agency responses more thoroughly, including by meeting with agencies directly (CalMatters).

For practitioners tracking governance and compliance, this disclosure matters because the law creates a formal reporting and oversight flow for systems that produce legally consequential outcomes. Civil rights, privacy, and civil liberties organizations advocated for the law, according to CalMatters, citing evidence that some automated decision tools produce disparate impacts in criminal justice, employment, and benefits adjudication. CalMatters highlights prior incidents linked to state systems, including an unemployment fraud detection process that paused benefits for 600,000 Californians around the 2020 holiday season, which underscores why advocates pressed for statutory oversight.

Whether the technology department publicly releases detailed inventories or risk assessments for the six systems named; whether agencies publish impact assessments or mitigation plans as required by the statute; and whether auditors or civil-liberties groups seek access to model documentation and procurement records. Observers will also watch if the state's deeper review process yields additional retroactive disclosures about legacy systems and vendor contracts.