Automation and evaluation of mutation testing for the new C++ standards

Abstract

Mutation testing is becoming increasingly widely used to evaluate the quality of test suites, especially to test programs coded in widely used programming languages in the industry. Mutation tools have arisen to automate the technique in different languages, including C++. With the increasing use of this technique, new mutation operators modeling possible faults often emerge to improve its abilities and adapt the tools to new advanced features. In this work, mutation operators for the new C++ standards, defined in previous work, are implemented and applied to generate and execute mutants in real programs. With this study, the MuCPP mutation tool is updated with the inclusion of these new operators. In addition, the improvements suggested in the definition of those operators can be finally tested, and conclusions about their utility in practice can be drawn. The implemented operators are checked on a set of four C++ programs that use these advanced features. The results show significant differences with the previous manual analysis: the number of invalid mutants was reduced by 64%, and we found fewer alive mutants (88%) and an increase in dead mutants (31%). In summary, both the number of mutants incorrectly classified in the previous manual analysis and the number of mutants generated (particularly equivalent mutants) have been reduced.