Multi-teacher distillation for pretraining a low-resource language

Abstract

Masked Language Models like BERT have brought a paradigm shift in the field of Natural Language Processing, achieving state-of-the-art results across numerous NLP tasks. However, pretraining such models for low-resource languages remains a considerable challenge. Existing approaches involve either training language-specific BERT models by collecting large text corpora or using massively multilingual models, which, unfortunately, face the challenge of per-language performance drop when handling multiple languages in a single model termed as ”curse of multilinguality”. A promising solution came in the form of MergeDistill (Khanuja et al., 2021), which can combine monolingual models from similar languages into a unified multilingual model using Knowledge Distillation and finds that the combined model performs better than each individual model. However, this approach remains unexplored for low-resource languages. Our research aims to fill this gap by adapting MergeDistill for the low-resource language Maltese. We aim to understand the impact of pretraining masked language models for such low-resource languages through the method of knowledge distillation from high-resource languages that are linguistically related. Our evaluation focuses on the Maltese language, involving knowledge distillation from a monolingual Maltese teacher model, as well as teachers from Arabic, English, and Italian. We analyze the performance differences among monolingual, massively multilingual, and language-specific multilingual models on both semantic and syntactic tasks. Our results indicate that knowledge distillation enhances the efficiency of pretraining, and the inclusion of related languages contributes positively to the evaluation tasks. Furthermore, these combined models retain most of their performance in evaluation tasks, despite encountering fewer pretraining data and being trained for less training time than monolingual models.