https://www.um.edu.mt/library/oar/handle/123456789/13902 2026-04-11T11:26:28Z 2026-04-11T11:26:28Z Paggio, Patrizia Mitterer, Holger Attard, Greta Vella, Alexandra https://www.um.edu.mt/library/oar/handle/123456789/145275 2026-04-01T07:38:58Z 2025-01-01T00:00:00Z

Title: Do hand gestures increase perceived prominence in naturally produced utterances? Authors: Paggio, Patrizia; Mitterer, Holger; Attard, Greta; Vella, Alexandra Abstract: This study investigates the effect of visually perceived gestures on the overall (multimodal) prominence of naturally occurring stimuli extracted from a multimodal corpus of Maltese conversations. Experiment participants were required to rate the prominence of target words in sentences presented to them as audiovisual and audio-only stimuli. In half of the stimuli, the target word was accompanied by a co-speech hand gesture. The results of the experiment show (i) that words produced with a co-speech gesture were rated as more prominent than words that were produced without one and (ii) that this was the case independently of whether raters could see those gestures (audiovisual condition) or not (audio-only condition). An acoustic analysis of the data shows that the presence of a co-occurring gesture has a significant effect on the pitch of the target vowel. The study suggests that gestures may provide the listener with an additional but not necessary cue to perceiving prominence.

2025-01-01T00:00:00Z Cafagna, Michele Rojas-Barahona, Lina M. van Deemter, Kees Gatt, Albert https://www.um.edu.mt/library/oar/handle/123456789/142795 2026-01-15T14:37:56Z 2023-01-01T00:00:00Z

Title: Interpreting vision and language generative models with semantic visual priors Authors: Cafagna, Michele; Rojas-Barahona, Lina M.; van Deemter, Kees; Gatt, Albert Abstract: When applied to Image-to-text models, explainability methods have two challenges. First, they often provide token-by-token explanations namely, they compute a visual explanation for each token of the generated sequence. This makes explanations expensive to compute and unable to comprehensively explain the model’s output. Second, for models with visual inputs, explainability methods such as SHAP typically consider superpixels as features. Since superpixels do not correspond to semantically meaningful regions of an image, this makes explanations harder to interpret. We develop a framework based on SHAP, that allows for generating comprehensive, meaningful explanations leveraging the meaning representation of the output sequence as a whole. Moreover, by exploiting semantic priors in the visual backbone, we extract an arbitrary number of features that allows the efficient computation of Shapley values on large-scale models, generating at the same time highly meaningful visual explanations. We demonstrate that our method generates semantically more expressive explanations than traditional methods at a lower compute cost and that it can be generalized to a large family of vision-language models.

2023-01-01T00:00:00Z Mariotti, Ettore Arias-Duart, Anna Cafagna, Michele Gatt, Albert Garcia-Gasulla, Dario Alonso-Moral, Jose Maria https://www.um.edu.mt/library/oar/handle/123456789/141214 2025-11-12T14:39:21Z 2024-01-01T00:00:00Z

Title: TextFocus : assessing the faithfulness of feature attribution methods explanations in natural language processing Authors: Mariotti, Ettore; Arias-Duart, Anna; Cafagna, Michele; Gatt, Albert; Garcia-Gasulla, Dario; Alonso-Moral, Jose Maria Abstract: Among the existing eXplainable AI (XAI) approaches, Feature Attribution methods are a popular option due to their interpretable nature. However, each method leads to a different solution, thus introducing uncertainty regarding their reliability and coherence with respect to the underlying model. This work introduces TextFocus, a metric for evaluating the faithfulness of Feature Attribution methods for Natural Language Processing (NLP) tasks involving classification. To address the absence of ground truth explanations for such methods, we introduce the concept of textual mosaics. A mosaic is composed of a combination of sentences belonging to different classes, which provides an implicit ground truth for attribution. The accuracy of explanations can be then evaluated by comparing feature attribution scores with the known class labels in the mosaic. The performance of six feature attribution methods is systematically compared on three sentence classification tasks by using TextFocus, with Integrated Gradients being the best overall method in terms of faithfulness and computational requirements. The proposed methodology fills a gap in NLP evaluation, by providing an objective way to assess Feature Attribution methods while finding their optimal parameters.

2024-01-01T00:00:00Z Assimakopoulos, Stavros https://www.um.edu.mt/library/oar/handle/123456789/140766 2025-10-31T12:53:12Z 2025-01-01T00:00:00Z

Title: Speech act theory : between narrow and broad pragmatics Authors: Assimakopoulos, Stavros Abstract: Speech act theory has been foundational in establishing pragmatics as an independent field of inquiry; yet, recent pragmatic research appears to have drifted away from the theoretical investigation of speech acts. This Element explores the reasons why this is so, focusing on the difference of perspective that emerges when the scope of the discipline is viewed through a narrow versus a broad lens. Following an overview of the initial exposition of speech act theory by Austin, it tracks its evolution, through subsequent Searlean and Gricean elaborations, to the currently received view. This view is then found to have diverged substantially from Austin’s original vision, largely due to its alignment with the narrow conception of pragmatics. Against this backdrop, it is suggested that embracing the broad take on the discipline can allow for a reintegration of Austin’s vision into the way we theorise about speech acts.

2025-01-01T00:00:00Z