Leveraging invariant prediction for mitigating specificity constraints in affect modelling

Abstract

Affect modelling aims to predict human emotional states from multimodal signals, yet current approaches often struggle to generalise beyond the specific datasets or contexts in which they are trained. This dissertation investigates the use of in variant features, predictors whose relationship with affective states remains stable across distinct environments, as a strategy to improve generalisability. To this end, two publicly available corpora, AGAIN and RECOLA, were systematically parti tioned into environments defined by user, task, and annotator triplets. An envi ronment refers to the conditions under which data is collected, and data gathered within the same environment is assumed to come from the same underlying distri bution. The Invariant Causal Prediction (ICP) framework was employed to identify stable features across these environments, which were then compared against full feature sets and principal components derived through PCA. Three supervised learning models—Logistic Regression, a feed-forward Neural Network,and a Long Short-Term Memory (LSTM)network — were trained under all three feature conditions, using group-based cross-validation to avoid information leakage. Results demonstrate that invariant features can deliver measurable benefits for feed-forward models, particularly in enhancing accuracy and correlation while substantially reducing feature dimensionality. However, their advantages were less consistent for sequence models like LSTMs, where temporal dependencies were not fully captured by invariants alone. Statistical significance tests further showed that invariant features improved balanced classification (F1) more strongly in AGAIN than in RECOLA,underscoring the dataset-specific nature of their effectiveness. Overall, the findings highlight both the promise and the limitations of invariance in affect modelling. While not a universal solution, invariant features represent a principled means of isolating robust predictors across heterogeneous contexts, contributing to the broader goal of developing affective systems that are reliable, interpretable, and adaptable across diverse real-world settings.