Metamorphic testing

Metamorphic testing (MT) is a property-based software testing technique, which can be an effective approach for addressing the test oracle problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected test cases or to determine whether the actual outputs agree with the expected outcomes.

Metamorphic relations (MRs) are necessary properties of the intended functionality of the software, and must involve multiple executions of the software. Consider, for example, a program that implements sin x correct to 100 significant figures; a metamorphic relation for sine functions is "sin (π − x) = sin x". Thus, even though the expected value of sin x₁ for the source test case x₁ = 1.234 correct to the required accuracy is not known, a follow-up test case x₂ = π − 1.234 can be constructed. We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency (after taking rounding errors into consideration) indicates a failure[1]: 31  of the program, caused by a fault[1]: 31  in the implementation.

MRs are not limited to programs with numerical inputs or equality relations. As an example, when testing a booking website, a web search for accommodation in Sydney, Australia, returns 1,671 results; are the results of this search correct and complete? This is a test oracle problem. Based on a metamorphic relation, we may filter the price range or star rating and apply the search again; it should return a subset of the previous results. A violation of this expectation would similarly reveal a failure of the system.

Metamorphic testing was invented by T.Y. Chen in a technical report [2] in 1998. Since then, more than 150 international researchers and practitioners have applied the technique to real-life applications. Some examples include web services,[3] computer graphics,[4] embedded systems,[5] simulation and modeling,[6] machine learning,[7] decision support,[8] bioinformatics,[9] components,[10] numerical analysis,[11] and compilers.[12] The first major survey of the field of MT was conducted in 2016.[13] It was followed by another major survey in 2018,[14] which highlights the challenges and opportunities and clarifies common misunderstandings.

Although MT was initially proposed as a software verification technique, it was later developed into a paradigm that covers verification, validation, and other types of software quality assessment.[15] MT can be applied independently, and can also be combined with other static and dynamic software analysis techniques such as proving and debugging.[16]

In August 2018, Google acquired GraphicsFuzz, a startup from Imperial College London, to apply metamorphic testing to graphics device drivers for Android smartphones.[17][18]

See also

• Mutation testing
• Property testing
• QuickCheck

References

1. 610.12-1990 - IEEE Standard Glossary of Software Engineering Terminology, IEEE, 1990, doi:10.1109/IEEESTD.1990.101064, ISBN 9781559370677
2. T.Y. Chen; S.C. Cheung; S.M. Yiu (1998), "Metamorphic testing: A new approach for generating next test cases", Technical Report HKUST-CS98-01 (PDF), Department of Computer Science, The Hong Kong University of Science and Technology, Hong Kong, arXiv:2002.12543.
3. C. Castro-Cabrera and I. Medina-Bulo, "An approach to metamorphic testing for WS-BPEL compositions", in Proceedings of the International Conference on e-Business, IEEE (2011).
4. R. Guderlei and J. Mayer, "Towards automatic testing of imaging software by means of random and metamorphic testing", International Journal of Software Engineering and Knowledge Engineering 17 (6): 757−781 (2007).
5. T.H. Tse, S.S. Yau, W.K. Chan, H. Lu, and T.Y. Chen, "Testing context-sensitive middleware-based software applications", in Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC '04), vol. 1, IEEE Computer Society, pp. 458−465 (2004).
6. S. Segura, R.M. Hierons, D. Benavides, and A. Ruiz-Cortes, "Automated test data generation on the analyses of feature models: a metamorphic testing approach", in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation (ICST '10), IEEE Computer Society, pp. 35−44 (2010).
7. J. Ding, X.-H. Hu, and V. Gudivada, "A machine learning based framework for verification and validation of massive scale image data", IEEE Transactions on Big Data (2017). doi: 10.1109/TBDATA.2017.2680460.
8. F.-C. Kuo, Z.Q. Zhou, J. Ma, and G. Zhang, "Metamorphic testing of decision support systems: A case study", IET Software 4 (4): 294−301 (2010).
9. L.L. Pullum and O. Ozmen, "Early results from metamorphic testing of epidemiological models", in Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom '12), IEEE Computer Society, pp. 62−67 (2012).
10. S. Beydeda, "Self-metamorphic-testing components", in Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC '06), vol. 1, IEEE Computer Society, pp. 265−272 (2006).
11. C. Aruna and R.S.R. Prasad, "Metamorphic relations to improve the test accuracy of multi precision arithmetic software applications", in Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI '14), IEEE (2014).
12. C. Lidbury, A. Lascu, N. Chong, and A.F. Donaldson, "Many-core compiler fuzzing", in Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '15), ACM, pp. 65−76 (2015).
13. S. Segura, G. Fraser, A.B. Sanchez, and A. Ruiz-Cortes, "A survey on metamorphic testing", IEEE Transactions on Software Engineering 42 (9}: 805-824 (2016).
14. T.Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T.H. Tse, and Z.Q. Zhou, "Metamorphic testing: A review of challenges and opportunities", ACM Computing Surveys 51 (1): 4:1-4:27 (2018).
15. Z.Q. Zhou, S. Xiang, and T.Y. Chen, "Metamorphic testing for software quality assessment: A study of search engines", IEEE Transactions on Software Engineering 42 (3): 264-28s4 (2016).
16. T.Y. Chen, T.H. Tse, and Z.Q. Zhou, "Semi-proving: an integrated method for program proving, testing, and debugging", IEEE Transactions on Software Engineering 37 (1): 109-125 (2011).
17. "GraphicsFuzz is acquired by Google". GraphicsFuzz. 6 August 2018. Retrieved 23 December 2020.
18. "Google buys GraphicsFuzz, bringing fuzzy logic and metamorphic testing to Android graphics drivers". Packt. 8 August 2018. Retrieved 27 December 2020.

External links

• "Metamorphic testing: Introduction and applications". ACM SIGSOFT Webinar.
• Sergio Segura; Zhi Quan Zhou (2018). ICSE 2018 technical briefing on metamorphic testing. International Conference on Software Engineering (ICSE 2018). doi:10.5281/zenodo.1256230.
• Zhi Quan (George) Zhou (9 April 2019). Automated testing of real-life self-driving systems and beyond: Keynote speech. IEEE International Conference on Artificial Intelligence Testing (AI Test 2019). San Francisco East Bay, CA, USA. doi:10.5281/zenodo.2636979.