AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

What they did

1. The work starts from a gap: most video-editing benchmarks only judge silent-video visuals or evaluate audio edits in isolation, missing real-world requests that need coordinated audio-visual changes.
2. The team curated 145 source videos and 196 human-verified audio-visually coupled editing instructions, then had humans validate 2,688 Yes/No checklist items that check both whether the edit was executed and whether untouched content was preserved.
3. Evaluation splits into four MLLM-judged dimensions—Instruction Following (IF), Fidelity Preserving (FP), Realism, and Editing Intent (EI, defined as IF x FP to capture full success)—plus seven automated metrics such as lip sync and audio-visual synchronization.
4. Guided by these diagnostics, they built AVE-Agent: a planner agent that breaks instructions into a dependency graph of video/audio/speech subtasks, an executor agent that runs and self-corrects each subtask, and an evaluator agent that rechecks the assembled clip and triggers remixing or replanning; it was compared against five existing systems (Wan2.7, HappyHorse, Gemini-Omni, Seedance, LTX2).

Findings

• AVE-Agent scored highest on Editing Intent and Instruction Following, with especially large gains on audio-side instruction following and audio-visual synchronization.
• LTX2 followed many instructions but largely regenerated the source video, giving it poor Fidelity Preserving; Gemini-Omni and Seedance often returned the original audio or video unchanged, inflating their preservation scores while failing to satisfy the instruction.
• Some models scored high on automated Video/Audio Aesthetic metrics but noticeably lower on the separate Realism rubric, showing technical quality and physical/logical plausibility diverge.
• Across five repeated scoring runs, automated metrics varied by less than 0.01 and MLLM-judge scores by less than 1%, and agreement with human annotations averaged close to 90% across the four dimensions.
• In the error analysis, AVE-Agent had the fewest total errors (175) versus 248-406 for baselines and showed no single dominant failure category, whereas Gemini's 144 audio-instruction-following errors and LTX2's 161 video-fidelity errors reflected non-response and over-regeneration respectively.

Where it can be used

• Using AVE-Compass to screen audio-video editing models before deployment for tasks that require synchronized picture-and-sound edits
• Diagnosing a specific model's characteristic failure pattern (e.g., over-regeneration vs. non-response) to guide pipeline fixes
• Referencing the plan-execute-evaluate agent design for accessibility-oriented workflows such as re-dubbing or content localization

Limits and open work

• The benchmark is deliberately kept at 145 videos / 196 instructions because scoring each clip via closed-source APIs and MLLM judging is expensive per instance.
• AVE-Agent depends on third-party tools whose behavior may change over time, which can affect long-term reproducibility of its results.
• Gemini missed 16 speech-related edits due to content moderation policies, so those cases are excluded from its reported results.
• AVE-Agent's automated Audio Similarity score is mildly depressed by a roughly 1-frame (about 8ms) muxing offset; the authors note that with sample-level alignment the audio is essentially preserved.
• The authors flag dual-use risks (deepfakes, non-consensual voice cloning) and note the released system ships without identity-cloning tools, recommending pairing it with provenance signals like watermarking.

Why it matters

Anyone evaluating or deploying audio-video editing models tends to judge quality by looking at the picture, but this work quantitatively shows that audio-side failures are the more prominent problem and gives a concrete diagnostic toolkit for it. It also gives practitioners specific failure categories to check for when choosing or building such editing systems.

Terms in this paper

• MLLM-as-Judge · Using a large multimodal language model to score edited outputs the way a human rater would
• checklist · A set of Yes/No questions verifying that an edit was executed correctly and that unedited content was preserved
• Editing Intent (EI) · A score computed as Instruction Following x Fidelity Preserving, capturing complete and clean edit execution
• subtask DAG · A dependency graph showing the order and cross-modal relations among the small editing steps derived from one instruction
• Fidelity Preserving (FP) · How well unedited visual and audio content is kept unchanged from the source video

Latest papers

• SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the timearXiv:2608.197992026-08-21
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papersarXiv:2608.197582026-08-21
• PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad actionarXiv:2608.198612026-08-21
• EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise itarXiv:2608.198912026-08-21
• EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknessesarXiv:2608.198802026-08-21
• Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM AgentsAI assistants would rather double-check facts than ask you a question, even when asking is the right callarXiv:2608.195642026-08-21
• SynFlow: A Multidimensional Diachronic Semantic Analysis ToolkitAn open-source tool that breaks down how a word's meaning changed, not just that it changedarXiv:2608.194722026-08-21
• Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based onearXiv:2608.195262026-08-21

Latest from METAL LAB

• Sakana AI Signs Deal With Japan's Defense Ministry for Intelligence Analysis AI Trial
• Hermes Agent builds its own skills the more you use it
• Is training AI on copyrighted books legal? Courts are still fighting it out
• Chinese gray market sells Anthropic Claude tokens at 10% of list price
• Even the Best AI Runaway Response Plan Among Five Major Labs Scores Only 3