JAV-CG The 1st International Workshop on Joint Audio-Video Comprehension and Generation

Towards Holistic and Physically Consistent Audio-Visual Generation

Generating rich audio-visual experiences requires integrating visible events, sounds beyond the camera’s view, and the physical relationships connecting what we see and hear. In this talk, I will present our recent work on multimodal generation, progressing from holistic audio generation to evaluating physical consistency in jointly generated audio and video. I will first introduce VinTAGe, which combines video and text conditioning to generate temporally synchronized on-screen and off-screen sounds while addressing modality bias. I will then present OmniSonic, which integrates speech with environmental sounds to enable more complete auditory scene synthesis. Finally, I will examine a fundamental question: do joint audio-video generation models understand physics? Through AV-Phys Bench, I will discuss how we evaluate physical commonsense within and across audio and video, and what current models reveal about the gap between perceptual plausibility and physical consistency. I will conclude with opportunities for building more controllable multimodal generative systems that faithfully capture the dynamics of the physical world.

Yapeng Tian is an Assistant Professor in the Computer Science Department at UT Dallas, where he leads the Computer Vision and Multimodal Computing Lab. His work has been recognized by the AAAI New Faculty Highlights, Cisco Faculty Research Award, and Amazon Research Award.

Audio-Visual Intelligence in Large Foundation Models: A Comprehensive Survey

Audio-Visual Intelligence (AVI) has emerged as a central frontier in artificial intelligence, bridging auditory and visual modalities to enable machines that can perceive, generate, and interact in the multimodal real world. In the era of large foundation models, joint modeling of audio and vision has become increasingly crucial, i.e., not only for understanding but also for controllable generation and reasoning across dynamic, temporally grounded signals. Recent advances, such as Meta MovieGen and Google Veo-3, highlight the growing industrial and academic focus on unified audio–vision architectures that learn from massive multimodal data. However, despite rapid progress, the literature remains fragmented, spanning diverse tasks, inconsistent taxonomies, and heterogeneous evaluation practices that impede systematic comparison and knowledge integration. This survey provides the first comprehensive review of AVI through the lens of large foundation models. We establish a unified taxonomy covering the broad landscape of AVI tasks, ranging from understanding (e.g., speech recognition, sound localization) to generation (e.g., audio-driven video synthesis, video-to-audio) and interaction (e.g., dialogue, embodied, or agentic interfaces). We synthesize methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization. Furthermore, we curate representative datasets, benchmarks, and evaluation metrics, offering a structured comparison across task families and identifying open challenges in synchronization, spatial reasoning, controllability, and safety. By consolidating this rapidly expanding field into a coherent framework, this survey aims to serve as a foundational reference for future research on large-scale AVI.

Survey paper

Hao Fei is a Senior Postdoctoral Researcher at the Department of Computer Science and the Big Data Institute, University of Oxford. His research focuses on natural language processing, computer vision, and multimodal learning, with an emphasis on large multimodal foundation models.

Multi-level Alignment in Audio-Visual Generation and Learning

How can machines connect what they see with what they hear to understand and generate coherent audio-visual experiences? This talk examines audio-visual alignment at multiple levels, ranging from semantic categories and object identities to the geometry and physical acoustics of a scene. I will present our recent work on three related problems. First, I will discuss audio-visual scene synthesis, where neural scene representations and physics-guided generative modeling enable spatial audio synthesis at novel source and listener positions. Second, I will describe sound separation guided by visual or language cues, including how pretrained audio generative models can isolate sounds without task-specific training. Third, I will introduce language-guided joint audio-visual editing, which modifies visual content and its accompanying sound while preserving source characteristics and maintaining consistency across modalities. Together, these studies show how the demands of audio-visual alignment change across tasks, from identifying a sound source to preserving content during editing and modeling how sound travels through a scene. I will discuss what these differences imply for designing models that produce semantically consistent and physically plausible audio-visual content.

Chenliang Xu is an Associate Professor of Computer Science at the University of Rochester. His research focuses on multimodal AI, integrating vision, sound, and language to understand and reason about dynamic scenes. He has coauthored 150 publications, and his work has received support from federal agencies including DARPA, ARPA-H, NIH, and NSF, as well as industry partners including Meta, Sony, Adobe, and Cisco. His honors include Rochester’s 2025 Edmund A. Hajim Outstanding Faculty Award, the James P. Wilmot Distinguished Professorship, and multiple paper and demonstration awards. He has served as an area chair for CVPR and ACM Multimedia and will serve as a Program Chair for WACV 2028. He received his Ph.D. from the University of Michigan in 2016.