OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization

Abstract

In recent years, the field of text-to-video (T2V) generation has made significant strides. Despite this progress, there is still a gap between theoretical advancements and practical application, amplified by issues like degraded image quality and flickering artifacts. Recent advancements in enhancing the video diffusion model (VDM) through feedback learning have shown promising results. However, these methods still exhibit notable limitations, such as misaligned feedback and inferior scalability.

To tackle these issues, we introduce OnlineVPO, a more efficient preference learning approach tailored specifically for video diffusion models. Our method features two novel designs, firstly, instead of directly using image-based reward feedback, we leverage the video quality assessment (VQA) model trained on synthetic data as the reward model to provide distribution and modality-aligned feedback on the video diffusion model. Additionally, we introduce an online DPO algorithm to address the off-policy optimization and scalability issue in existing video preference learning frameworks. By employing the video reward model to offer concise video feedback on the fly, OnlineVPO offers effective and efficient preference guidance.

Extensive experiments on the open-source video-diffusion model demonstrate OnlineVPO as a simple yet effective and more importantly scalable preference learning algorithm for video diffusion models, offering valuable insights for future advancements in this domain.

Motivation

Video Feedback Learning

We analyze and summarize the limitations of current video feedback learning methods as follows:

Method

OnlineVPO - Video-Centric & Online Preference Optimization

We propose OnlineVPO, a feedback learning method with more :

The differences between our method and other methods are summarized as follows:

Experiment

Quantitative Comparison

We report the performance comparison with other methods on VBench:

Qualitative Comparison

We showcase some visual comparison example videos:

BibTeX

@misc{zhang2024onlinevpoalignvideodiffusion,
      title={OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization}, 
      author={Jiacheng Zhang and Jie Wu and Weifeng Chen and Yatai Ji and Xuefeng Xiao and Weilin Huang and Kai Han},
      year={2024},
      eprint={2412.15159},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.15159}, 
}