Evaluation of alternative design choices for evolutionary mutation testing by means of automated configuration

Abstract

Mutation testing is a well-established but costly technique to assess and improve the fault detection ability of test suites. This technique consists of introducing subtle changes in the code of a program, which are expected to be detected by the designed test cases. Among the strategies conceived to reduce its cost, evolutionary mutation testing (EMT) has been revealed as a promising approach to select a subset of the whole set of mutants based on a genetic algorithm (GA). However, like any other metaheuristic approach, EMT’s execution depends on a set of parameters (both classical of GAs and context-specific ones), so different configurations can greatly vary its performance. Currently, it is difficult to clarify what are the best values for those parameters by applying manual parameter tuning and whether new design choices could improve its effectiveness with other combinations of values. The experience carried out in this paper applying iterated racing, a well-known automated configuration algorithm, reveals that EMT's performance has been undervalued in previous studies; the new configuration found by iterated racing was able to enhance EMT’s results in all C++ object-oriented programs used in the experiments. This study also confirms alternative design choices as convenient options to improve EMT in this context, namely, detecting and penalizing equivalent mutants by means of Trivial Compiler Equivalence, and learning which mutation operators produced live mutants in the past generations.