Perceptually Motivated Guidelines for Voice Synchronization in Film

Actors performing emotional scripts.

People

CMU Robotics Institute

Disney Research, Pittsburgh

Abstract

We consume video content in a multitude of ways, including in movie theaters, on television, on DVDs and Blu-rays, online, on smart phones, and on portable media players. For quality control purposes, it is important to have a uniform viewing experience across these various platforms. In this work, we focus on voice synchronization, an aspect of video quality that is strongly affected by current post-production and transmission practices. We examined the synchronization of an actor’s voice and lip movements in two distinct scenarios. First, we simulated the temporal mismatch between the audio and video tracks that can occur during dubbing or during broadcast. Next, we recreated the pitch changes that result from conversions between formats with different frame rates. We show, for the first time, that these audio visual mismatches affect viewer enjoyment. When temporal synchronization is noticeably absent, there is a decrease in the perceived performance quality and the perceived emotional intensity of a performance. For pitch changes, we find that higher pitch voices are not preferred, especially for male actors. Based on our findings, we advise that mismatched audio and video signals negatively affect viewer experience.

Paper Front Page

Elizabeth Carter, Lavanya Sharan, Laura Trutoiu, Iain Matthews, Jessica K. Hodgins, Perceptually Motivated Guidelines for Voice Synchronization in Film, in ACM Transactions on Applied Perception, 7(4), Article 23, July 2010.

[PDF (2.9 MB)] [Video (MOV, 49.3 MB)] 

 

Content on this page requires a newer version of Adobe Flash Player.

Get Adobe Flash player

 

This work was supported in part by NSF Grant 0811450 and a Microsoft Research Grant to J.K. Hodgins.

* These authors contributed equally to this work.