Automated email customer service : a local, low-resource IT scenario

Abstract

Email is one of most prevalent communication channels used by customers to request a service. Automated email classification can allow companies to provide more efficient and effective service. For this research, we collect and carry out classification experiments on realworld data from MITA’s service call centre. We analyze the performance of three text representation techniques, namely a TF-IDF based approach, Word2vec and GloVe, in conjunction with traditional machine learning classifiers. Since MITA uses a three-tier service hierarchy to label email requests, we assess the viability of both flat and hierarchical classification. We also fine-tuned BERT for text classification and compared it to the traditional machine learning classifiers. Moreover, we assess the effects of partial layer freezing to reduce training time while still retaining a suitable level of performance. Our study shows that fine-tuning BERT provides the best results within our IT support ticket classification scenario. However, a support vector machine classifier in conjunction with TF-IDF-based text vectors provides comparable results while requiring significantly less training time and compute resources.