Augmented storytelling : bringing drawing to AR

Abstract

Humans can naturally interpret 3D shapes from simple 2D sketches, even when lacking depth or perspective—a skill that’s predominantly noticeable in children’s drawings. This project aims to replicate that intuition by transforming children’s 2D sketches into interactive, textured 3D models within an Augmented Reality (AR) environment. Specifically focusing on interpreting character drawings which can then be edited in the AR environment to create a live interactive story. Unlike existing methods that ignore colour and texture, this system retains the full visual identity of the original sketch [1, 2]. A custom PaintApp captures both the image and stroke data. Using an inflation method, the 2D silhouette is ”puffed up” into a 3D mesh that reflects the original intent [2]. The final model is imported into Unity and displayed via HoloLens 2, where users can interact and modify it using Constructive Solid Geometry (CSG) tools for real-time editing. The project combines Python and C# to manage data processing and 3D reconstruction, while Unity serves as the primary platform for AR integration. The PaintApp captures sketches and processes them using inflation-based mesh generation. This is then exported into Unity and viewed on HoloLens 2. Real-time editing is enabled using a CSG library, giving users control over their 3D creations in a playful and intuitive way. This work not only contributes to the domain of sketch-based modelling but also promotes open-ended creativity by removing the limitations of flat 2D expression—turning every sketch into an immersive 3D experience.