Investigating ways to mitigate the oracle problem using HCI techniques

Abstract

The Oracle Problem is the problem that given a test case, one should determine whether the system behaved well or not. In the industry, so far, the human testers decide whether a system has worked as expected or not by either pre-programming a test case with the system's expected outcome or else by manually looking at the system working and analysing in real time whether the system worked as expected. This is only possible due to the human oracle having tacit domain knowledge. This, therefore, puts us in a position to analyse the cognitive behaviour of the human as an oracle. Brain Computer Interfacing, which is a specialised area in Human Computer Interaction, helps us analyse the cognitive behaviour of the human by recording EEG data which represents the electrical activity in the brain. The hypothesis that drove this work is that HCI, and in particular, BCI techniques, can be leveraged to help mitigate the Oracle Problem. By presenting a series of test cases to human subjects, this thesis attempted to study the link between what the participants report explicitly and their emotional state (via EEG monitors). Since this dissertation is exploratory in nature, the methodology proceeded in iterations, having four in total. The rst was an exploratory study to delve into BCI and EEG techniques, the second was a pilot study where new data was collected and analysed, the third was mainly data collection at a testing conference in London and the last one was data collection in a controlled environment. Since the second iteration was a pilot study, after the data was collected, various analysis techniques were tried to nd the one which gives the most wishful results. During the iterations, I consulted three experts in the BCI eld for feedback about my protocol before reaching conclusions. The data gathered during the iterations highlighted above was fed into a kNN classi fier for classifying mental states. From this data, two types of classi cation models were built - a global model (i.e. a classi cation model which includes the data for all participants) and an individual model (i.e. a classi cation model built for each participant). When comparing the results from the global and individual models through distribution analysis, they turned out to be considerably di fferent. Moreover, when comparing the results from the data gathered in normal lab environment vs the data gathered in a controlled environment were also considerably di fferent.