Automated Research Assessment Reveals Misaligned Funding Criteria

The paper audits the Brazilian Research Productivity (PQ) Grant evaluation run by CNPq using an interpretable machine learning pipeline applied to researcher CVs and OpenAlex bibliometrics. The authors convert policy dimensions into measurable variables and test each regulatory criterion as an empirical hypothesis. Their models achieve high discrimination, with mean AUC 0.96, while showing that the effective evaluative signal is concentrated in a small feature subset.

The study operationalizes dimensions such as bibliographic production, human resource training, and scientific recognition from structured CV fields and OpenAlex records. It uses a block-based adaptation of the Boruta feature selection procedure to probe variable importance across several standard classifiers. The analysis emphasizes interpretability over black-box performance, isolating features that consistently improve classification of grant levels, with a focus on identifying Level 1A researchers.

• •Key predictive features identified: bibliographic production, graduate-level supervision, institutional management roles.

This work reframes evaluation criteria as testable inputs rather than normative axioms, applying interpretable ML to a public funding regime. The high AUC shows that PQ labels carry a robust statistical structure, which is useful for automation and audit. The more important finding is the mismatch: many officially emphasized criteria do not contribute measurably to classification. For practitioners, the paper is a blueprint for evidence-driven audits of research metrics and automated decision systems in science policy.

The study raises governance questions for agencies using automated or semi-automated assessment. Follow-ups should test causal links, extend to other funding programs and countries, and evaluate how disclosure of the compact evaluative signal changes applicant behavior and policy design.