Online Emotion-Driven Generation of Multiple Appropriate Facial Reactions

The multiple appropriate facial reaction online generation task aims to generate real-time, appropriate, and diverse facial reactions for virtual listeners in response to audio-visual behaviours expressed by a human speaker. While recent approaches have focused on improving reaction diversity and coarse synchronicity, they often fail to capture emotionally coherent responses that align with both the emotion type and intensity level of the speaker. In this work, we propose an emotion-driven framework that treats the speaker’s emotional state as the core driving force behind listener behavior. Our framework integrates a pre-trained audio emotion encoder (PAEE) and visual emotion encoder (PVEE) to extract fine-grained emotional representations from speech and facial expressions. We further design a lightweight, online-capable Motion Representation Module (MRM), optimized for real-time generation, that captures emotional intensity through facial motion amplitude and variation, enabling our system to dynamically modulate the strength of listener reactions with low latency. Besides, an Unpredictable Motion Generator (UMG) further introduces minor, stochastic perturbations, making the generated reactions more lifelike and individualized. Extensive experiments demonstrate that our method achieves significant improvements in reaction appropriateness and diversity, while maintaining real-time performance. The codes are available at this link.