Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods

详细信息    查看全文

• 作者：Andre Takeshi Endo ; ^{aendo@icmc.usp.br} ; Adenilso Simao ^{adenilso@icmc.usp.br}
• 关键词：Model based testing ; Test automation ; Experiments ; Test case generation method ; Mutation testing ; Conformance testing
• 刊名：Information and Software Technology
• 出版年：2013
• 出版时间：June, 2013
• 年：2013
• 卷：55
• 期：6
• 页码：1045-1062
• 全文大小：1681 K

文摘

Context

Testing from finite state machines has been investigated due to its well-founded and sound theory as well as its practical application. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods.

Objective

We compare the methods that generate complete test suites from finite states machines. The test suites produced by the W, HSI, H, SPY, and P methods are analyzed in different configurations.

Method

Complete and partial machines were randomly generated varying numbers of states, inputs, outputs, and transitions. These different configurations were used to compare test suite characteristics (number of resets, test case length) and the test suite length (i.e., the sum of the length of its test cases). The fault detection ratio was evaluated using mutation testing to produce faulty implementations with an extra state.

Results

On average, the recent methods (H, SPY, and P) produced longer test cases but smaller test suites than the traditional methods (W, HSI). The recent methods generated test suites of similar length, though P produced slightly smaller test suites. The SPY and P methods had the highest fault detection ratios and HSI had the lowest. For all methods, there was a positive correlation between the number of resets and the test suite length and between the test case length and the fault detection ratio.

Conclusion

The recent methods rely on fewer and longer test cases to reduce the overall test suite length, while the traditional methods produce more and shorter test cases. Longer test cases are correlated to fault detection ratio which favored SPY, though all methods have a ratio of over 92 % .