Please use this identifier to cite or link to this item:
Title: The myth of reproducibility : a review of event tracking evaluations on Twitter
Authors: Mamo, Nicholas
Azzopardi, Joel
Layfield, Colin
Keywords: Artificial intelligence
Machine learning
Data collection platforms
Information retrieval
Issue Date: 2023-04-05
Publisher: Frontiers Research Foundation
Citation: Mamo, N., Azzopardi, J. & Layfield, C. (2023). The myth of reproducibility: A review of event tracking evaluations on Twitter. Frontiers in Big Data, 6, 1067335.
Abstract: Event tracking literature based on Twitter does not have a state-of-the-art. What it does have is a plethora of manual evaluation methodologies and inventive automatic alternatives: incomparable and irreproducible studies incongruous with the idea of a state-of-the-art. Many researchers blame Twitter's data sharing policy for the lack of common datasets and a universal ground truth–for the lack of reproducibility–but many other issues stem from the conscious decisions of those same researchers. In this paper, we present the most comprehensive review yet on event tracking literature's evaluations on Twitter. We explore the challenges of manual experiments, the insufficiencies of automatic analyses and the misguided notions on reproducibility. Crucially, we discredit the widely-held belief that reusing tweet datasets could induce reproducibility. We reveal how tweet datasets self-sanitize over time; how spam and noise become unavailable at much higher rates than legitimate content, rendering downloaded datasets incomparable with the original. Nevertheless, we argue that Twitter's policy can be a hindrance without being an insurmountable barrier, and propose how the research community can make its evaluations more reproducible. A state-of-the-art remains attainable for event tracking research.
Appears in Collections:Scholarly Works - FacICTAI

Files in This Item:
File Description SizeFormat 
Frontiers_Mamo2023.pdf360.94 kBAdobe PDFView/Open

Items in OAR@UM are protected by copyright, with all rights reserved, unless otherwise indicated.