Methodological pitfalls in predictive mutation testing: threats, impact and open challenges

Abstract

Predictive Mutation Testing (PMT) has emerged as a promising technique for reducing the high computational cost of traditional mutation testing. This is achieved by using predictive models to estimate mutant behavior without actually requiring their execution. However, despite its growing interest, PMT is still a young and evolving field and, therefore, its experimental studies remain particularly vulnerable to various methodological threats that may compromise the validity, comparability, and reproducibility of results. This paper aims to identify and examine key methodological threats in prior PMT studies that not only affect their validity, but may also limit the technique’s ability to reach its full predictive potential. For each of the eight identified threats, spanning the main stages of the PMT workflow, we describe its nature, analyze how it has been addressed (or overlooked) in previous work, and offer a validation list of recommended practices. We further conduct an empirical validation to substantiate the impact of these threats. In addition, based on the gaps identified in this analysis, we outline several open challenges that remain unexplored in the field, such as the lack of standardization in dataset sharing, the prevalence of Java-centric studies with method-level operators, and the challenges posed by class imbalance and project heterogeneity. This work contributes to strengthening the methodological soundness of PMT research, promoting more meaningful cross-study comparisons, and encouraging the adoption of practices that foster reproducibility, scalability, and real-world applicability.