The Prevention and Observability Frontier in AI Action Auditing

Abstract

An artificial intelligence audit can both reveal a defective action and prevent that action from being released. Consequently, the audit sample changes the residual loss that it is intended to estimate. This paper develops a finite-population framework for this dual role of verification. A model-assisted estimator targets the loss remaining in the actual unaudited batch, and an exact error identity distinguishes this random target from average policy risk. The framework yields a finite-sample upper confidence bound, an impossibility result for unauditable actions, and a convex allocation rule balancing predicted prevention against worst-case estimation variance. In a transparent synthetic population of 6,000 actions, five policies receive the same expected budget of 1,200 audits. Under a concealed error-rate shift, the joint allocation reduces expected residual loss by 34.7% and estimation root mean squared error by 89.3% relative to risk-priority auditing. Under aligned scores, however, it incurs 34.0% more residual loss than risk-priority auditing, exposing the cost of observability. Twenty thousand audit replications per policy and scenario verify the analytical calculations and reveal conservative confidence bounds. The contribution is an explicit connection between corrective auditing, the statistical target, and resource allocation, rather than a new inverse-probability estimator. The results support auditable AI release decisions under stated assumptions, without claiming empirical validation on deployed systems.