{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/02c2e406-61a8-421e-bd5d-5278f2185e6b","identifier":"02c2e406-61a8-421e-bd5d-5278f2185e6b","url":"https://forgecascade.org/public/capsules/02c2e406-61a8-421e-bd5d-5278f2185e6b","name":"OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation","text":"# OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation\n\n**Authors:** Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu, Hu Yu\n**arXiv:** https://arxiv.org/abs/2605.12480v1\n**Published:** 2026-05-12T17:56:59Z\n\n## Abstract\nRecent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audio-video generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: (i) multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; (ii) multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; (iii) uniform credit assignment, where fine-grained cross-modal alignment regions fail to get efficient exploration. These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results. To address these challenges, we propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations: (1) Modality-wise advantage routing, which routes independent per-reward advantages to their respective modality generation branches. (2) Layer-wise gradient surgery, which selectively detaches video-branch gradients on shallow audio layers while retaining those for cross-modal interaction layers. (3) Region-wise loss reweighting, which modulates policy optimization toward critical regions related to audio-video synchronization and fine-grained alignment. Extensive experiments on JavisBench and VBench with the LTX-2 backbone demonstrate that OmniNFT achieves comprehensive improvements in audio and video perceptual quality, cross-modal align","keywords":["cs.CV","cs.AI"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"},"dateCreated":"2026-05-13T06:00:09.440000Z","dateModified":"2026-05-13T06:00:09.440000Z"}