Predicting stock price trends : integrating technical indicators and news sentiment analysis

Abstract

Forecasting the stock market has long been an exciting yet challenging research topic due to its characteristic volatile and chaotic nature. The aim of similar studies remains to maximise trading profits whilst minimising risk. This dissertation investigates whether Machine Learning (ML) and Deep Learning (DL) models can support short-term trading in the Standard & Poor’s 500 (S&P 500) index. First, a published study predicting next-day closing prices of the index is replicated and extended using an enhanced feature set including historical index prices, technical indicators, smoothing functions, line-of-best-fit gradients, and sentiment signals derived from Google Trends. Eight ML and DL architectures are optimised through hyperparameter tuning and evaluated using multiple input feature sets. This work recognises that prediction accuracy is important, however, it gives more weight to whether these predictions can drive a profitable trading strategy, including direct comparisons with a buy-and-hold strategy. Although Directional Accuracy (DA) across regression models approximates random performance, the trading strategies derived from these predictions yield informative results. The top-performing regression-based trading strategy, informed by the Support Vector Regressor (SVR) model, achieves a trading return of 17.78%, compared to the 27.51% S&P 500 return. However, the remaining models perform poorly, with three out of seven models incurring losses. These results motivated a shift in the predictive framework to a classification task, in which models forecast next-day trading-position entries. This reformulation leads to improved trading returns across seven out of eight models, with the best-performing Artificial Neural Network (ANN)-based strategy yielding a return of 22.91%, an increase of five percentage points relative to the regression-based strategies. Statistical significance tests, in particular the Mann-Whitney U test, confirm these improvements. Despite these gains in trading performance, none of the Artificial Intelligence (AI)-driven trading strategies were able to outperform the S&P 500 index. Differences between validation- and test-set performance suggest the presence of concept drift, highlighting the ever-evolving market dynamics. Overall, this study demonstrates that reframing the prediction task can strengthen trading outcomes; however, this also highlights the challenges faced when predicting the stock market.