Page-turning when playing an instrument is annoying at best. Page-turning from digital music is even more so. Although there are tablet-based apps for page-turning specifically for musicians, these are either based on timed-scrolling, that is, the page scrolls at a fixed time, through tapping with your finger on the tablet screen, through tapping on a special foot pedal or, by using sound recordings, whereby the audio signal is matched to its location on the written music score.
Dr Alexandra Bonnici, a lecturer with the Department of Systems and Control Engineering, and a piano player, investigated some of the apps available. She found that timed-scrolling apps were not very useful, because they did not take into account the fact that the player can vary the speed with which they play. Tapping on the screen is closest to page-turning on a printed book. But because the tablet screen is typically smaller than the A4 page, more page turns are needed on a tablet than on a printed page. Also, the tablet is not always responsive to fast taps, so the page may not turn immediately. The pianist would also need to lift their hand from the piano to turn the page. The audio-based app that was tested was very erratic - it performed well when a single melody line was played correctly but the performance deteriorated when accompanying bass was added, or when some notes were played incorrectly.
Alexandra hypothesised that since the pianist looks at the music sheet to play the score, then by tracking the pianists point-of-gaze, it would be possible to predict when the pianist needs to turn the page for the pianist. To investigate this hypothesis, Alexandra turned to her colleagues, Dr Stefania Cristina and Prof. Kenneth Camilleri who both have considerable experience on eye-gaze tracking. Together we developed a proof-of-concept for an eye-gaze music page turner which was reported in the International Symposium on Document Engineering in 2017 (here). This made use of an infra-red eye-gaze tracking device and promised good results - it out-performed the apps we tested it against in terms of its ability to turn pages. Since turning a page means that the previous page is no longer accessible, we adjusted the way the music is displayed effectively splitting the page in half and updating one half of the screen with new material while keeping the half the pianist is looking at the same.
Our initial prototype, however, was sensitive to movements of the pianists head and to eye-glances away from the screen (e.g. if one were to look at fellow musicians, at a conductor or at ones own hands). And so, we proposed this project as a final year project for the academic year 2018-2019. The project was taken up by André Tabone. André started learning the piano at a young age and despite stopping formal tuition in his teens, the interest in music remained. This project proposal, therefore, caught his attention. For his final year project, André captured recordings of five pianists as they played some specifically selected piano pieces. Using these recordings, André was able to design a Kalman filter model which uses the eye-gaze data obtained from the infra-red sensor to obtain a better estimate of the pianist's point of regard. This work greatly improves the performance of our page-turner making it more robust to glance away from the score.