Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systems

arXiv:2512.146292026-08-17

A new benchmark checks whether AI music editors accidentally wreck the parts of a song you didn't want changed

Music editing AI can change one thing (like genre or timbre) but often messes up rhythm, melody, or structure without meaning to. This paper builds MuseCPBench, the first standardized benchmark to measure this 'Music Context Preservation' across four musical facets. Testing five popular editing systems shows none of them reliably preserve everything, and each has different weak spots.

What they did

Problem: existing music editing systems (e.g. MusiConGen, InstructME, MS-Diffusion, ZETA) each check different, inconsistent things when claiming they preserve the original music, making fair comparison impossible.
Solution: MuseCPBench organizes what should stay unchanged into four categories - harmony, rhythm & meter, structural form, and melodic content & motifs - with specific measurable metrics for each.
Method: five representative editing systems (MusicGen, MusiConGen, MusicMagus, ZETA, RefinPaint) were tested on real music/MIDI datasets, comparing edited output against the original using these metrics, plus a 'Vanilla' baseline that generates music without any editing instruction.
Result: RefinPaint (which refines existing MIDI rather than generating from scratch) preserved context best overall. MusicMagus kept tonal color well but nearly destroyed rhythm. MusicGen struggled most with timing/rhythm. ZETA kept rhythm well but lost melodic motifs.
Finding: strong performance on one musical facet doesn't guarantee strong performance on another - no single system excelled everywhere, revealing a general blind spot in current music editing AI.

Table 1: Comparison of music generation and editing systems by editing type, MCP evaluation, and evaluated facets.
Method	Backbone	Task	Edit	MCP Evaluation?	Facets
MusicGen [3]	Transformer	Text / Music-conditioned generation	—	✗	—
MusiConGen [8]	Transformer	Text / Music-conditioned generation	Harmony, Rhythm	✓	Harmony, Rhythm
AUDIT [18]	Diffusion	Instruction-guided editing	Local attributes	✗	—
InstructME [6]	Diffusion	Instruction-guided editing	Harmony, Rhythm	✓	Harmony, Rhythm
MusicMagus [20]	Diffusion	Music inversion	Global structure	✗	—
RefinPaint [15]	Diffusion	Iterative inpainting	Local segments	✗	—
MS-Diffusion [11]	Diffusion	Joint generation and separation	Stems (sources)	✓	Stem Fidelity
ZETA [10]	Diffusion	Editing and inversion	Structure	✓	Structure (human-judged)

Table 2: MCP performance across music editing baselines. Bold: the best result. Underlined: the second-best result.
Baseline	Harmony & Tonality	Rhythm & Meter
	Circle of Fifth-distance ↓	Chroma DTW Similarity ↑	Major Minor Score ↑	Δ BPM ↓	Beat F-measure ↑	Information Gain ↑
Vanilla	0.333	0.860	0.188	26.638	0.229	0.141
MusicGen	0.197	0.870	0.334	36.690	0.273	0.391
MusiConGen	0.210	0.902	0.474	0.000	0.492	0.362
MusicMagus	0.068	0.955	0.738	13.262	0.030	0.029
ZETA	0.177	0.945	0.441	4.125	0.732	0.604
RefinPaint	0.009	0.988	0.943	8.825	0.949	0.951

Why it matters

For anyone building or using AI music editing tools, this gives a concrete, comparable way to check if an edit tool ruins the rhythm or melody while changing style - something that was previously unmeasured. It also gives model builders a diagnostic map of exactly where their architecture (autoregressive vs diffusion vs refinement-based) tends to fail.

Terms in this paper

Music Context Preservation (MCP) · the ability of a music editing AI to keep the parts of a song unchanged that weren't meant to be edited
MuseCPBench · the benchmark introduced here to measure MCP consistently across systems
harmony / rhythm & meter / structural form / melodic content & motifs · the four musical dimensions the benchmark checks are preserved after editing
circle-of-fifths distance · a music-theory based measure of how close two musical keys are to each other
Beat F-measure / Information Gain · metrics for how well the timing of beats in edited music matches the original
Adjusted Rand Index (ARI) · a score measuring how well two segmentations of a song's sections agree with each other
autoregressive generation · generating audio step-by-step in sequence, which can drift or accumulate errors over longer pieces

Original abstract (English)

Music editing plays a vital role in modern music production, with applications in film, broadcasting, and game development. Recent advances in music editing systems have enabled diverse editing tasks such as timbre transfer, instrument substitution, and genre transformation. However, many existing works overlook evaluating their ability to preserve musical facets that should remain unchanged during editing, which we define as Music Context Preservation (MuseCP). While some studies do consider MuseCP, their evaluation protocols and metrics are not comprehensive. To address this, we introduce the first MuseCP evaluation framework, MuseCPEval, that covers four categories of music facets with fine-grained and well-tailored metrics to capture nuanced changes in music attributes. Objective validation and a human study demonstrate the effectiveness of these metrics. Moreover, the case studies on diverse music editing systems illustrate the practical utility of these metrics as a testbed and diagnostic tool, providing insights into the strengths and limitations of existing systems. We hope our metrics and findings can offer practical guidance for developing more effective and reliable music editing strategies with strong MuseCP capability

Authors · Yash Vishe

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Yash Vishe et al., arXiv:2512.14629, CC BY 4.0