A multimodal causal-inspired approach towards affect modeling

Abstract

This study introduces a novel causal-inspired approach to affect modeling that leverages distribution shifts and emotional subjectivity as advantages rather than obstacles. Our methodology employs causal graph discovery algorithms to identify relationships between affect measurements and emotional states across different environments, defined by user-task-annotator combinations. We implement constraint-based (PC) and score-based (GES) algorithms to discover environment-specific causal graphs for constructing Bayesian networks for affect prediction. The approach is validated on the RECOLA dataset, demonstrating competitive performance with neural networks and random forests. A key finding reveals a consistent directed edge between arousal and valence nodes, extending Russell’s circumplex theory by identifying conditional dependencies when affect measurements are explicitly observed. This work contributes by integrating causal inference in affect modeling, implementing multiple graph discovery approaches, validating on a benchmark dataset, and establishing groundwork for developing causal models of affect.