Project Overview

Co-speech gestures are an essential part of face-to-face communication. This project studies audio-driven dyadic co-speech gesture generation: given the speech of two participants in a conversation, the goal is to generate natural, synchronized, and interactive full-body motion for both people.

Instead of following the diffusion-based approach commonly used in recent dyadic gesture generation, I transfer ideas from text-to-motion generation—discrete motion tokens and iterative masked generation—to the audio-driven two-person setting.

Method

The framework contains three main components. A 2D VQ-VAE encodes continuous full-body motion into discrete motion tokens. A pretrained WavLM extracts speech representations from both participants, which are aligned and fused as conditioning signals. A Masked Transformer then models the joint token sequence of both participants and progressively reconstructs masked motion tokens. Audio conditions are injected through linear projection and Adaptive Layer Normalization (AdaLN).

Iterative Masked Inference

During inference, the model starts from masked motion tokens and progressively predicts and refines them over multiple iterations. At each step, low-confidence predictions can be masked again and regenerated, allowing the model to gradually construct the final motion sequence.

For sequences longer than the 150-frame training window, I introduce a sliding-window autoregressive inference strategy. Overlapping windows and prefix conditioning carry motion context from one window to the next, enabling coherent generation of arbitrarily long conversations while reducing discontinuities at window boundaries.

Key Contributions

• Adapted discrete motion representation and masked generative modeling to audio-driven dyadic co-speech gesture generation.
• Designed a two-speaker audio-conditioning pipeline using WavLM and AdaLN, while jointly modeling both participants in a unified motion-token sequence.
• Developed a sliding-window autoregressive inference scheme for stable long-sequence generation, together with Step Unroll training and Classifier-Free Guidance.

Video Results

Example motion sequences generated by the proposed model.

Limitations & Future Work

Current automatic metrics do not fully capture interaction-specific qualities such as response timing, behavioral complementarity, and coordination between participants. Some generated clips also exhibit limited motion amplitude or weak interaction cues. Future work could explore interaction-aware evaluation metrics, richer semantic conditioning, and stronger physical or controllability constraints.