https://www.um.edu.mt/library/oar/handle/123456789/8368 Previously known as Department of Intelligent Computer Systems Sat, 22 Aug 2026 04:21:27 GMT 2026-08-22T04:21:27Z https://www.um.edu.mt/library/oar/handle/123456789/148548 Title: From measurement to mitigation : exploring the transferability of debiasing approaches to gender bias in Maltese language models Authors: Galea, Melanie; Borg, Claudia Abstract: The advancement of Large Language Models (LLMs) has transformed Natural Language Processing (NLP), enabling performance across diverse tasks with little task-specific training. However, LLMs remain susceptible to social biases, particularly reflecting harmful stereotypes from training data, which can disproportionately affect marginalised communities.We measure gender bias in Maltese LMs, arguing that such bias is harmful as it reinforces societal stereotypes and fails to account for gender diversity, which is especially problematic in gendered, low-resource languages.While bias evaluation and mitigation efforts have progressed for English-centric models, research on low-resourced and morphologically rich languages remains limited. This research investigates the transferability of debiasing methods to Maltese language models, focusing on BERTu and mBERTu, BERT-based monolingual and multilingual models respectively. Bias measurement and mitigation techniques from English are adapted to Maltese, using benchmarks such as CrowS-Pairs and SEAT, alongside debiasing methods Counterfactual Data Augmentation, Dropout Regularization, Auto-Debias, and GuiDebias. We also contribute to future work in the study of gender bias in Maltese by creating evaluation datasets.Our findings highlight the challenges of applying existing bias mitigation methods to linguistically complex languages, underscoring the need for more inclusive approaches in the development of multilingual NLP. Wed, 01 Jan 2025 00:00:00 GMT https://www.um.edu.mt/library/oar/handle/123456789/148548 2025-01-01T00:00:00Z https://www.um.edu.mt/library/oar/handle/123456789/148547 Title: MELABenchv1 : benchmarking large language models against smaller fine-tuned models for low-resource Maltese NLP Authors: Micallef, Kurt; Borg, Claudia Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across various Natural Language Processing (NLP) tasks, largely due to their generalisability and ability to perform tasks without additional training. However, their effectiveness for low-resource languages remains limited. In this study, we evaluate the performance of 55 publicly available LLMs on Maltese, a low-resource language, using a newly introduced benchmark covering 11 discriminative and generative tasks. Our experiments highlight that many models perform poorly, particularly on generative tasks, and that smaller fine-tuned models often perform better across all tasks. From our multidimensional analysis, we investigate various factors impacting performance. We conclude that prior exposure to Maltese during pre-training and instruction-tuning emerges as the most important factor. We also examine the trade-offs between fine-tuning and prompting, highlighting that while fine-tuning requires a higher initial cost, it yields better performance and lower inference costs. Through this work, we aim to highlight the need for more inclusive language technologies and recommend for researchers working with low-resource languages to consider more “traditional” language modelling approaches. Wed, 01 Jan 2025 00:00:00 GMT https://www.um.edu.mt/library/oar/handle/123456789/148547 2025-01-01T00:00:00Z https://www.um.edu.mt/library/oar/handle/123456789/148546 Title: Investigating adapters for parameter-efficient low-resource automatic speech recognition Authors: Samin, Ahnaf Mozib; Nayak, Shekhar; De Marco, Andrea; Borg, Claudia Abstract: Recent years have witnessed the adoption of parameter-efficient adapters in pre-trained language models for natural language processing. Yet, their application in speech processing remains less studied. In this work, we explore the adapters for low-resource speech recognition, introducing a novel technique - ConvAdapt into pre-trained speech models. We investigate various aspects such as data requirements, transfer learning within adapters, and scaling of feed-forward layers in adapters. Our findings reveal that bottleneck adapters offer competitiveness with full fine-tuning with at least 10 hours of data, but they are not as effective in few-shot learning scenarios. Notably, ConvAdapt demonstrates improved performance in such cases. In addition, transfer learning in adapters shows promise, necessitating research in related languages. Furthermore, employing larger speech models for adapter-tuning surpasses fine-tuning with ample data, potentially due to reduced overfitting than fine-tuning. Wed, 01 Jan 2025 00:00:00 GMT https://www.um.edu.mt/library/oar/handle/123456789/148546 2025-01-01T00:00:00Z https://www.um.edu.mt/library/oar/handle/123456789/148545 Title: Medical prognosis from electronic health records in Spanish Authors: Lebeña, Nuria; Borg, Claudia; Casillas, Arantza; Pérez, Alicia Abstract: Predicting future diagnostic codes from Electronic Health Records (EHRs) is a complex task due to the unstructured nature of medical texts. This study introduces RiskProg, a system designed to predict future diagnoses from Spanish EHRs. It aims to structure EHRs through generative LLMs and then predict future diagnoses by highlighting the most influential words that contributed to the prediction. RiskProg utilizes a three-step approach: (1) a summarizer that structures patient visit information into comprehensive summaries, (2) a future potential diagnoses predictor, and (3) a salient word identifier based on integrated gradients to highlight the textual elements influencing predictions. The system was evaluated using the Osa dataset, which includes 26,731 annotated Spanish discharge summaries. The incorporation of summarized EHR documents enhanced classification accuracy, with expanded summaries outperforming the baseline classification with raw documents. Explainability, achieved through word-level attributions, increased clinician trust by providing insights into the model’s decision-making process. RiskProg showcases the potential of structuring unstructured clinical data for next diagnosis prediction. The approach offers both an explanation of the predicted codes and structured summaries that make it easier to assess patient admission information. Wed, 01 Jan 2025 00:00:00 GMT https://www.um.edu.mt/library/oar/handle/123456789/148545 2025-01-01T00:00:00Z