VideoAuto-R1 vs Qwen3-VL-8B on VideoMME and LongVideoBench

[DeCai01]

Thanks for the good work and extensive analysis. I found the accuracy of VideoAuto-R1 on VideoMME and LongVideoBench is lower than the baseline of Qwen3-VL-8B, is there any analysis or insights? Thanks

[sming256]

Thanks for your interest in our work. We think this is mainly due to three reasons:

1. Limited room for improvement on perception-oriented benchmarks.
   Qwen3-VL-8B is already strong on perception-heavy video QA (e.g., VideoMME / LongVideoBench). Our RL methodology primarily targets reasoning and self-correction, so gains are larger on reasoning-heavy benchmarks (e.g., VideoMMMU, MVP), while improvements on perception-dominant tasks can be smaller or fluctuate slightly.

2. The Qwen baseline is sensitive to implementation, while VideoAuto-R1 is more robust.
   We observed that Qwen3-VL-8B results can vary across different PyTorch/Transformers implementations. Under provided GitHub environment, the baseline drops more, while VideoAuto-R1 remains stable and becomes stronger than the baseline.

Model	Implementation	VideoMME	LongVideoBench
Qwen3-VL-8B	Internal	72.5	67.6
VideoAuto-R1	Internal	71.7	67.4
Qwen3-VL-8B	GitHub	70.8	65.7
VideoAuto-R1	GitHub	71.2	67.5

3. RL training focuses on verifiable reasoning data.
   Our RL fine-tuning uses a smaller, verification-friendly reasoning dataset with fewer long-video/perception-oriented QA samples than Qwen3-VL pretraining. Thus, minor fluctuations on perception benchmarks are expected, while we consistently improve reasoning-heavy benchmarks with better efficiency.

We hope this clarifies your concerns.