Optical estimation of facial muscle activity for emotion detection : validation against electromyography and real-time prototype evaluation

Abstract

PURPOSE: This paper proposes and validates a novel three-stage framework for non-invasive emotion detection based on the transformation of facial video recordings into biosignal-like time series analogous to surface electromyography (EMG). The study aims to investigate the extent to which optically derived facial signals can serve as reliable proxies for physiological muscle activity and support real-time emotion recognition without the need for direct sensor contact.

DESIGN/METHODOLOGY/APPROACH: The research was conducted in three consecutive stages. In Stage 1, concurrent facial video and surface EMG recordings were collected from five participants exposed to emotionally valenced stimuli. Optical signals were extracted using the FLAME morphable face model and Facial Action Coding System (FACS)-based descriptors and compared with EMG ground-truth measurements across four facial muscle channels. Cross-modal correspondence was evaluated using Pearson correlation analysis and temporal alignment procedures. In Stage 2, the validated optical signal extraction pipeline was implemented within a remote-capable data collection application, enabling contactless acquisition and processing of facial-expression data. In Stage 3, the methodology was integrated into a portable real-time prototype system designed for automatic emotion recognition and tested on twenty participants.

FINDINGS: The results demonstrate that facial video data can be transformed into meaningful biosignal-like representations corresponding to underlying muscular activity. The strongest optical–EMG correspondence was observed for the Zygomaticus major muscle (r = 0.733 ± 0.188), while moderate correlations were obtained for the Corrugator supercilii (r = 0.282 ± 0.147) and Orbicularis oculi (r = 0.239 ± 0.216) muscles. Frontalis measurements exhibited greater variability (r = 0.239 ± 0.309). The analysis identified a systematic electromechanical delay of approximately 0.2 seconds between physiological and optically derived signals. The final prototype successfully classified ten discrete emotional categories in real time across twenty consecutive participants, confirming the feasibility of the proposed approach.

PRACTICAL IMPLICATIONS: The proposed framework offers a scalable and non-invasive alternative to traditional physiological emotion-monitoring techniques. Potential applications include human–computer interaction, affective computing, healthcare monitoring, psychological assessment, adaptive educational systems, customer experience research and wearable or mobile technologies requiring real-time emotion recognition without physical sensors.

ORIGINALITY/VALUE: The paper introduces an innovative methodology that bridges computer vision and physiological signal analysis by converting facial video streams into EMG-like representations of muscle activity. Its originality lies in the validation of optical–EMG correspondence, the identification of temporal synchronization characteristics and the successful implementation of a portable real-time emotion recognition system. The study contributes to the development of contactless affective computing technologies and provides a foundation for future research on physiologically informed emotion detection methods.