Quality in interlingual AI dubbing : exploring machine and human intersections

Abstract

The emergence and increasing prevalence of AI-driven end-to-end dubbing solutions necessitate an investigation into the quality of the generated output. This article offers an exploration of the intersections between machine output and human intervention in the production of AI-driven dubs, while identifying the nature of necessary human actions within the current state of the art. To this end, the article shares insights drawn from two independent exploratory studies carried out in training settings. First, an English AI dub, generated for didactic purposes, was evaluated and rated against its official studio-recorded counterpart. This served as a preliminary pilot application of the Script, Speech, and Sound (SSS) quality assessment framework (Spiteri Miggiani, 2024) across diverse training environments. Second, another experiment evaluated, rated, and compared the output of three different Italian AIdubbed versions of the same original clip, generated by three different platforms. The studio version was also evaluated as a benchmark. Issues were identified and labeled, and the specific “actions” required from human linguists were identified and categorized. While acknowledging that these findings are tool-dependent, context-specific, and subject to change as technology evolves, both cases nevertheless reveal consistent patterns where machine outputs intersect with human linguistic judgment. The key areas where human expertise appears to be essential include synchronization, voice consistency and expressiveness, narrative and semiotic cohesion, sound design, and language and translation.