A framework to support test tool design and acquisition

Abstract

Software testing is an important facet of software delivery, supported by tools intended to improve the efficiency and effectiveness of testing. Industry experience and academic research show that tool adoption can be problematic; tools are acquired but not used, or are used but do not deliver. The research problem this thesis addresses is how to design tools that better match the needs of testers to operate in an increasingly complex socio‐technical environment. Industry practitioners’ and experts’ experiences with tools were explored, through in‐depth interviews, workshops and surveys. It was found that testers experienced frustrations arising from tools which, while offering attractive interfaces, did not provide quality in use necessary to meet testers’ needs. In this work, this is referred to as the ‘illusion of usability’. This illusion arises from a superficial understanding of usability as being focused on the user interface, working with a limited persona set, and focusing narrowly on usability, without considering the other attributes that make up quality in use. Furthermore, finding that testers do not conform to the stereotype of IT workers, and cannot be represented in tool design by a simple, small set of personas or archetypes, it was decided to apply an HCI lens to the problem, with the research question “How can HCI techniques help with the design of test tools?” In answering this question, this work proposes an empirically grounded framework (idea‐t), which supports decision making in both design and acquisition of tools through a set of heuristics, guidelines and activities. The idea‐t framework (“Influencing the Design, Evaluation and Acquisition of Tools for Testing”) emerged following a series of studies and was iteratively reviewed and validated through five industry case studies. Learning was carried forward from each case study and applied to the framework. The five formative case studies iteratively informed the development of the framework, while also providing evidence of its effectiveness in the process. Participants reported benefits including new insights and improved communication within their teams. A final retrospective analysis evaluated the framework by examining a backlog of customer issues raised on a commercial tool; it was found that potentially 40% of issues could have been mitigated by the idea‐t framework. Expert reviews were also carried out to assess the latest version of the framework, where experts from testing, test tool development, and HCI provided positive feedback on the framework’s efficacy, and suggestions for its practical application.