Trevor Darrell||Chris Bregler
Interval Research||Interval Research and UC Berkeley
Intelligent Analysis/Synthesis of Facial Imagery
The intelligent processing of images of faces is a key resource
in building interactive systems with natural and intuitive interfaces.
In this presentation we describe two different systems for the
analysis/synthesis of facial imagery, carried out by ourselves
and colleagues at Interval Research.
The first system, the Magic Morphin Mirror, is an example of
real-time, on-line vision processing to detect faces and drive
an interactive graphics display. We use multi-modal integration
of stereo, color, and pattern cues to robustly segment people
and track their faces in crowded, open-background environments.
Our first application was an interactive virtual mirror, demonstrated
at SIGGRAPH '97, where a user's facial expression was distorted
into a surreal effect in real-time.
The second system, Video
Rewrite, is an off-line, photo-realistic facial animation
system that uses existing footage to create automatically new
video of a person mouthing words that she did not speak in the
original footage. The system reorders the mouth images in
the training footage to match the phoneme sequence of the new
audio track. A new automatic morphing technique is used to combine
these mouth gestures into the final video sequence. The
resulting video combines the dynamics of the original actor's
articulations with the mannerisms and setting dictated by the
background footage.
These systems and their future extensions hold out the promise
of creating fully synthetic characters from video examples, and
animating them using real-time, untethered human performance.
This talk describes joint work with Malcolm Slaney, Michele Covell,
Gaile Gordon, John Woodfill, Mike Harville and others at
Interval Research.