Multi-agent communication optimization in large language models (LLMs)

Abstract

What if these LLMs could converse among themselves? What if they could solve problems independently?” If that were the case, we wouldn’t need to be involved in the brainstorming part of a conversation; we could make a final decision on whether or not to use their conclusions. This curiosity led to the creation of a multi-agent system (mas) since humans currently always need to be in a conversation with an LLM so that they don’t drive the conversation down the wrong path. This issue, commonly referred to as the Degeneration-of-Thought (DoT) problem, limits the effectiveness of LLMs in tasks that require multi-step reasoning and dynamic dialogue. Traditional approaches, such as Chain of Thought and Tree of Thought, have mitigated some challenges but remain restricted by the single-agent model. Another issue why humans always need to be in a conversation is so that the LLMs don’t work with wrong information, this issue is known as hallucination. So we decided to add more agents into the conversation, whose roles could be to oversee a conversation or keep it on the correct path, hoping that it helps mitigate these issues. One could say this stems from the popular saying ”two heads are better than one.” This thesis addresses the DoT problem by introducing a new Multi-Agent System (MAS) that enables multiple LLMs to work together, producing richer, more diverse, and meaningful conversations. The system was developed in two phases: version one laid the groundwork for structured multi-agent discussions, while version two introduced advanced features such as agent turn-taking mechanism, agent autonomy and integrated tools for handling specialized tasks, such as calculation or fact-checking. To assess the system’s performance, both quantifiable key performance indicators (KPIs) such as participation rates and turn-taking balance, along with qualitative metrics like coherence and innovation, were evaluated. The results indicated that version two significantly enhanced dialogue efficiency, achieving a near-perfect turn-taking balance (0.95) and faster response times. For the qualitative assessment, metrics were developed by where both humans and LLM evaluated the same dialogues, with the goal of determining if LLM could serve as a reliable evaluator. The evaluations from both human participants and the AI demonstrated strong coherence and concreteness, though generating innovative ideas remains an area needing improvement. Due to the close correlation between human and LLM ratings, large language models were subsequently employed to evaluate dialogue feedback using the same metrics applied to the systems-generated dialogue The system was also compared to other SOTA like AutoGen and MetaGPT amongst others. Based on practical applications and results shown When compared, the proposed system demonstrated greater flexibility and capability in handling complex tasks. This work provides a solid foundation for future advancements in collaborative dialogue systems, offering a more adaptive and efficient approach to tackling the DoT problem, and broadening the potential applications of LLMs in multi-agent settings.