A comparative study of technical and creative text translation : evaluating the performance of ChatGPT

Abstract

In recent years, there has been a growing interest in using generative artificial intelligence (AI), such as ChatGPT, for machine translation (MT) tasks. One notable feature of generative AI systems is that users can provide instructions to guide the output. This capability presents both opportunities and challenges in the field of Translation Studies. Therefore, this study aims to evaluate the quality of MTs generated by ChatGPT (GPT-3.5). The impact of prompting techniques on translation quality is also analysed in order to gain insights into optimising the use of generative AI in translation workflows. The study employed three distinct prompts – zero-shot prompt, context prompt, and few-shot with context prompt – to translate both a technical and a creative task for the English-German language pair. The raw MT outputs were manually annotated based on the DQF-MQM (Dynamic Quality Framework - Multidimensional Quality Metrics) framework, and the results of these annotations were then compared. The analysis revealed that the majority of errors concern style. The translations produced using the simple zero-shot prompt outperform those generated with the other two more complex prompts. However, the longer prompts led to a steady reduction in stylistic errors, while accuracy errors increased. This dissertation demonstrates that ChatGPT can generate coherent document translations when using a simple prompt.