Prompt scaffolding : guiding non-experts to limit hallucinations in large language model responses

Abstract

This paper investigates whether guided scaffolding limits hallucination and improves output consistency in LLM summarization tasks performed by non-expert users. As LLMs become more accessible to the public, understanding interface designs that support accurate information generation is essential for minimizing hallucination and improving user experience. This research employs an experimental design comparing two groups. A control group using self generated prompts without guidance, and an experimental group receiving structured scaffolding to guide prompt construction. Participants in the control group use AI tools to summarise texts with minimal guidance, reflecting natural usage patterns. This approach establishes baseline performance for typical non-expert AI users who rely on intuition rather than systematic strategies when prompting AI tools. Participants in the experimental group follow a structured workflow with three stages. Pre-reading analysis to identify key facts and terms, guided prompt construction using a fill-in template, and verification asking AI to check for hallucinated information. Participants will complete a short fact-based text summarization task conducted remotely via Google Forms. This methodology allows for controlled comparison while maintaining ecological validity through remote participation. Output quality will be measured using a three-dimensional scoring rubric that evaluates factual accuracy, coverage of important information, and hallucination rate. The weighted scoring system prioritizes factual accuracy and hallucination prevention over comprehensiveness. Dual weighting (0-4 points) is applied to the accuracy and hallucination dimensions, while single weighting (0-2 points) applies to comprehensiveness, yielding a maximum overall score of 10 points per participant. This weighting scheme emphasizes the primary research objectives of minimizing factual errors and fictional content. The research is based on two main hypotheses that the guided prompt scaffolding reduces the incidence of factual errors and hallucination in LLM-generated summaries by non-expert users, and that the guided prompt scaffolding reduces variance in summary quality across non-experts, producing more consistent outputs regardless of individual prompting skill. If the hypotheses are supported, empirical evidence will be obtained that a simple UI-level scaffolding mechanism can limit hallucinations and factual errors of summaries generated using LLM while reducing performance variance across users, enabling concrete design recommendations for consumer LLM interfaces. Furthermore, this research establishes a framework for evaluating prompt-guided mechanisms. If the hypotheses are not supported, valuable insights into potential barriers to the scaffolding effect, such as cognitive load or individual differences in user engagement, will be gained, contributing to future intervention design.