Test Data Reuse for the Evaluation of Continuously Evolving Classification Algorithms Using the Area under the Receiver Operating Characteristic Curve

Performance evaluation of continuously evolving machine learning algorithms presents new challenges, especially for high-risk application domains such as medicine. In principle, to obtain performance measures that generalize to a target population, a new independent test dataset randomly drawn from the target population should be used each time a new performance evaluation is required. However, test datasets of sufficient quality are often hard to acquire, and it is tempting to utilize a previously used test dataset for a new performance evaluation. With extensive experiments on simulated and real data we illustrate how such a “naive” approach to test data reuse can inadvertently result in overfitting the algorithm to the test data, resulting in a generalization loss and overly optimistic conclusions about the algorithm performance. We investigate the use of a modified version of the reusable holdout mechanism of Dwork et al. [Science, 349 (2015), pp. 636--638], which allows for repeated reuse of the same test dataset. We extend their approach to the use of AUC, the area under the receiver operating characteristic curve, as the reported performance metric. Theoretical guarantees for our method are proven to hold in extremely data-rich scenarios. However, our empirical results indicate promising performance of the proposed technique even on small data. With extensive simulation studies and experiments on real medical imaging data we show that our procedure indeed substantially reduces the problem of overfitting to the test data, even when the test dataset is small, at the cost of a mild additional uncertainty on the reported test performance.