Tip Of AI
Discover what you can do with AI
MultiTalk is a framework designed to generate videos of multiple people engaging in conversation, driven by synchronized audio streams and guided by textual prompts. It can take reference images of characters and multi-channel audio inputs to produce coherent and lifelike interactions. The framework features an additional audio cross-attention layer that conditions the generation process on audio inputs, ensuring precise lip sync and natural motion. It introduces the Label Rotary Position Embedding (L-RoPE) technique to correctly bind each audio stream to the corresponding character. Training strategies include partial parameter updates and multi-task objectives that preserve instruction-following capabilities and improve generalization to different video styles such as cartoons, singing performances, and instructional dialogues. Extensive evaluations on talking head, talking body, and multi-person datasets demonstrate that MultiTalk outperforms existing single-person and multi-stream methods in visual quality and audio-video alignment. The model generates consistent lip movements, realistic body gestures, and clear conversational flow. The code and pre-trained models are available for academic research only, and commercial use is not permitted.