“To trust a LIAR”: does machine learning really classify fine-grained, fake news statements?

Abstract

Fake news refers to deceptive online content and is a problem which causes social harm. Early detection of fake news is therefore a critical but challenging problem. In this paper we attempt to determine if state-of-the-art models, trained on the LIAR dataset can be leveraged to reliably classify short claims according to 6 levels of veracity that range from “True” to “Pants on Fire” (absolute lies). We investigate the application of transformer models BERT, RoBERTa and ALBERT that have previously performed significantly well on several natural language processing tasks including text classification. A simple neural network (FcNN) was also used to enhance each model’s result by utilising the sources’ reputation scores. We achieved higher accuracy than previous studies that used more data or more complex models. Yet, after evaluating the models’ behaviour, numerous flaws appeared. These include bias and the fact that they do not really model veracity which makes them prone to adversarial attacks. We also consider the possibility that language-based, fake news classification, on such short statements is an ill-posed problem.