VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

arXiv:2608.186072026-08-18

영상과 소리를 같이 만드는 AI, 어떤 결과물이 더 나은지 사람처럼 판단하는 채점관을 만들었다

영상과 음성을 동시에 생성하는 AI 모델을 강화학습으로 개선하려면 좋고 나쁨을 판단할 채점 기준(보상 신호)이 필요한데, 기존에는 화질·음질·싱크 등을 따로따로 점수 매겨 합치다 보니 사람이 보기엔 어색해도 점수만 높은 결과물이 나오는 문제가 있었다. 연구팀은 사람이 직접 두 영상 중 더 나은 것을 고르고 그 이유를 남긴 대규모 데이터셋(VAPref-10K)을 만들고, 이를 바탕으로 단계적 학습을 거쳐 사람처럼 이유를 대며 판단하는 채점 모델 VA-Judger를 개발했다. 이 채점 모델을 보상으로 써서 영상-음성 생성 모델(LTX-2)을 다시 학습시키자 사람 평가에서 기존 방식보다 훨씬 높은 선호도를 얻었다.

논문 원문 초록 (영문)

Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.

저자 · Yinming Huang

그림 출처: Yinming Huang et al., arXiv:2608.18607, CC BY 4.0