VR comm : communication in VR using LLM

Abstract

Virtual Reality (VR) spaces are designed to simulate realistic situations in a virtual environment. Designing agents that support and enhance realistic human‐machine interaction in such virtual environments thus becomes a crucial goal. This Final Year Project (FYP) takes into consideration the creation of a VR chatbot with enhanced communication capabilities using a Large Language Model (LLM). One problem which is found in research is that most chatbots lack the capabilities of natural interaction such as those between humans to humans. In a VR environment these challenges extend to lip syncing, eye blinking and facial expressions together with coherent and meaningful text or speech communication are non‐trivial and are constantly being experimented with to make agents and bots as human‐like as possible. This FYP has reviewed a number of technologies that have been used for a number of years as well as those that are currently being developed. Several case studies in different sectors such as healthcare, marketing and education on Artificial Intelligence (AI) chatbots and VR environments were also reviewed to help gain more insights into the design of communication‐based bots. Some examples include VR‐Enabled Engineering chatbots and VR chatbots to reduce emotional stress. For this FYP, the theories of presence in VR were used to inform the design process to give the illusion of stable spatial presence and the illusion of social interaction. The ethical issues regarding AI based chatbots and VR environments were taken in consideration and integrated in the experiment design process. Unity was chosen to be the game engine used to create a VR based chatbot with Generative Pre‐ Trained Transformer (GPT) models being the LLMs to be used for the chatbot integration, openAI’s whisper‐1 for Speech To Text (STT) transcriptions and Meta’s WitAI for Text To Speech (TTS) Natural Voices. GPT‐3.5‐0613 Turbo and GPT‐4.0 Turbo were chosen as the two LLMs that were compared for text and speech communication. A loudness detector script was used for lip syncing, making use of the loudness variable of the voice to determine the scale with which the lips on the avatar would move. The evaluation of this study compares which models pair best with each other to create a more realistic experience. A group of participants were asked to communicate with each GPT‐model based chatbot for a duration of time. Conversations were analysed qualitatively whilst a survey investigating the communication experience with the avatar inside the VR provided more insights into the usability and realism of the model.