Fine-grained topic detection and tracking on Twitter

Abstract

With its large volume of data and free access to information, Twitter revolutionised Topic Detection and Tracking (TDT). Thanks to Twitter, TDT could build timelines of real-world events in real-time. However, over the years TDT struggled to adapt to Twitter’s noise. While TDT’s solutions stifled noise, they also kept the area from building granular timelines of events, and today, TDT still relies on large datasets from popular events. In this paper, we detail Event TimeLine Detection (ELD) as a solution: a real-time system that combines TDT’s two broad approaches, document-pivot and feature-pivot methods. In ELD, an on-line document-pivot technique clusters a stream of tweets, and a novel feature-pivot algorithm filters clusters and identifies topical keywords. This mixture allows ELD to over come the technical limitations of traditional TDT algorithms to build fine-grained timelines of both popular and unpopular events. Nevertheless, our results emphasize the importance of robust topic tracking and the ability to filter subjective content.