Leveraging P2P systems to address the test scenario explosion problem

Abstract

Modern software development is characterised by a strong customer focus and electronic delivery mechanisms which make it very easy for customers to buy and install a vendor’s software. However, it also makes it very easy for customers to buy and install software from competing vendors and as such it is more important than ever for deployed software to be as correct and bug-free as possible. Whilst certain types of testing can be done in the lab with a high degree of confidence that results will hold when the software is deployed in the wild, in reality software systems are subject to influence from whatever environment they end up being deployed in. Varying factors in customer environments can include operating systems, services packs, device drivers, network connectivity, resource usage by other software, and so on. Any variation or combination of these factors can lead to a situation where a system deviates from its expected behaviour. The problem is amplified even further on mobile devices whereby devices can move between different networks, interrupt apps for phone calls, have varying screen sizes, have user interference in the form of turning features on and o↵ to preserve battery power, vendor-specific operating system code, and so on. A conservative calculation indicates that a software system can be subjected to tens of thousands of different scenarios. Even if one were to execute just one test case against each scenario, obtaining any form of realistic coverage is infeasible for even the most resource-rich organisations. Cloud and grid infrastructures have been proposed as solutions to improving the scalability of software testing since 2008. However, we argue that cloud computing systems are too homogenous and are not representative of real-world usage scenarios. We refer to this problem as the Test Scenario Explosion Problem.