Comparing anomaly detection algorithms for trading behaviour on the Malta Stock Exchange

Abstract

The difficulty in interpreting irregular trading activity in a small and relatively illiquid market is under-represented in academic literature, where statistical abnormality may reflect structural liq uidity rather than genuine unusual behaviour. This dissertation aims to investigate and compare the usefulness of selected unsupervised anomaly detection algorithms for identifying irregular trading behaviour on the Malta Stock Exchange (MSE). Particularly, the specific aim was to assess whether selected anomaly detection methods can generate meaningful and interpretable signals within the specific environment of the Maltese market. A pragmatic mixed-method approach was adopted. The quantitative component used historical daily trading data for the seven most liquid equities over the period 2010–2025 and compared four unsupervised detection algorithms: Isolation Forest (IF), Local Outlier Factor (LOF), DBSCAN, and Expectation-Maximisation (EM). The evaluation framework combined score-based compari son, event-based validation using public company announcements, and qualitative interpretation through several semi-structured expert interviews. The findings show that the models captured different forms of irregularity. Isolation Forest produced the clearest anomaly ranking and emerged as the primary anomaly detection method; LOF identified different local irregularities; EM revealed broader latent trading regimes; and DBSCAN was more useful to understand market and density structure than for standalone anomaly detection. Event-based validation revealed that a significant share of the strongest anomaly instances occurred near public announcements, whereas the interviews highlighted the importance of contextual interpretation and professional judgement. This dissertation establishes that unsupervised anomaly detection algorithms can provide useful support for surveillance on the MSE but should be treated as a contextual support tool rather than as direct evidence of market abuse.