Assessing cognitive workload during software engineering activities

Abstract

Assessing the difficulty of a task as it is being performed can provide substantial value to an organisation. Fritz et al. posit that such a capability would enable developers to, amongst other benefits, revise the estimates for tasks, reduce bug counts, or even offer support to developers where needed. The authors further propose a methodology for assessing task difficulty among developers, using physiological data from eye trackers, brain activity (electroencephalography or EEG), and electrodermal activity (EDA). Although the study by Fritz et al. yielded positive results, one cannot help but notice that the devices used in their research required a specialised lab setup and were arguably intrusive. This project set out to investigate the extent to which the results obtained in the aforementioned study could be replicated by exclusively using sensors provided by a commercial off-the-shelf smart watch. The intention here was to explore the possibility of rendering such work more accessible in an industry setting. Following a review of commercially available devices, the Fitbit Sense watch was chosen for the purpose of this study. The methodology followed by Fritz et al. was adapted to allow the use of heart-rate sensors on the said device. Twenty participants were asked to complete a set of software development tasks that were designed in a way that each successive task induced more cognitive stress on the participant. Time limits were also imposed to regulate the length of the study and induce additional pressure. Data was collected by means of a smartphone app during the exercise as it extracted the data from the watch worn by the participants. As in Fritz et al., a Bayes classifier was used to classify windows of heart-rate data as ‘stressful’ or ‘not stressful’. To train this classifier, the participants were asked to complete the widely used NASA-TLX questionnaire about each task immediately after completing the respective tasks thereby providing a subjective workload rating and perceived difficulty for each task, which was subsequently used in conjunction with heart-rate windows as training data for the classifier. Despite the varying results, it was possible to register a precision level of 77.3%, thus demonstrating that real-time task-difficulty assessment using non-invasive commercial off-the-shelf devices is possible. This conclusion presents ample opportunities for future research in the area, ranging from improving the classification methodology, to encompassing the real-time task-difficulty assessment as part of a set of real-world productivity tools that could support knowledge workers in their day-to-day jobs.