CMBMeTest: Generation of Test Suites Using Model-Based Testing Plus Constraint Programming and Metamorphic Testing

Abstract

Various software testing techniques have been shown to be successful in producing high-quality test suites for software where the code is not accessible (black-box approach). Nevertheless, no method has been found to guide combining some of these in a general way. In this study, a test suite generation method for black-box software called CMBMeTest was created to respond to these challenges. It employs several coupled software testing techniques, namely, model-based testing (MBT), constraint programming (CP), and metamorphic testing (MT). CMBMeTest provides step-by-step instructions for using the information available (such as program specifications, inputs and outputs) to create an initial test suite that covers the model obtained, using a combination of MBT and CP (referred to as MBT+CP). Furthermore, using the metamorphic relations (MRs) of MT, a better test suite was produced from that initial test suite. The method allows particular stages to be iterated to improve the results by building new models and new MRs. A comprehensive case study was conducted, employing CMBMeTest to produce encouraging results. Mutation testing was used to evaluate the test suite, and the first round produced a high mutation score. A more detailed model was used to repeat the process, with similar outcomes.