An Analysis of Class Imbalance Challenges in Predictive Mutation Testing

Abstract

This study evaluates the efficacy of Predictive Mutation Testing (PMT) models, focusing on the impact of preprocessing, class rebalancing, and model selection on predictive accuracy. By filtering unreached mutants, artificially inflated metrics caused by trivial correlations were minimized, enhancing the ability of the model to identify complex patterns. While ensemble models like CatBoost and XGBoost showed high effectiveness in detecting killed mutants, their performance in identifying survived mutants was limited, underscoring the need for refined strategies to address class imbalance. The study further identifies that previous PMT research often lacks class-specific metrics, such as the Matthews Correlation Coefficient, essential for assessing balanced performance. This oversight can yield misleading evaluations by favoring the majority class, which may falsely indicate high model reliability. Although SMOTE rebalancing improved recall for survived mutants, balancing sensitivity and precision remains challenging. The findings advocate for the development of more balanced datasets in future PMT research to enhance both accuracy and robustness in real-world testing scenarios.