Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
arXiv:2608.198632026-08-19
오디오 스펙트로그램을 '다음 조각 맞히기'로만 학습시켜도 최고 성능이 나온다
NAPE는 소리를 잘게 자른 조각(패치)들을 순서대로 보여주면서 다음 조각의 특징값을 예측하게 학습시키는 방법이다. 복잡한 복원 장치나 교사-학생 구조 없이 인과적 마스킹과 그래디언트 차단만으로 학습이 되도록 설계했다. 여섯 개 오디오·음성 벤치마크에서 여러 과제 최고 성능을 냈고 모델 크기를 키울수록 성능도 꾸준히 좋아졌다.
무엇을 했나
소리를 로그멜 스펙트로그램(소리를 시간-주파수 그림으로 표현한 것)으로 바꾼 뒤 작은 사각형 조각들로 잘라 순서를 매기고, 앞선 조각들만 보고 다음 조각의 특징값을 맞히도록 트랜스포머를 학습시켰다
복원 디코더, 별도의 음향 토크나이저, 이동평균 교사 모델, 부가 손실 함수 같은 기존 기법들을 전부 빼고 인과적 마스킹(미래 조각을 못 보게 가리는 장치)과 스톱그래디언트(정답 쪽으로는 학습 신호가 안 가게 막는 장치)만 사용했다
조각을 나열하는 순서(래스터, 타임메이저, 지그재그, 대각선)를 네 가지로 비교했고, 시간 축을 따라 진행하는 래스터·지그재그·대각선 방식이 주파수를 먼저 다 훑는 타임메이저보다 뚜렷하게 나았다
AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP 등 여섯 개 벤치마크에서 여러 과제 최고 수준(파인튜닝 기준) 성능을 냈고, 19M/85M/303M 파라미터의 세 크기 모두에서 크기가 커질수록 성능이 꾸준히 향상됐다
가중치를 고정한 채 분류기만 학습하는 선형 프로빙에서도 좋은 결과를 냈고, 지도학습 없이도 어텐션 패턴이 음향적으로 의미 있는 구조를 스스로 형성했다
Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddings z are processed by the causal encoder h and predictor g to produce predictions z^, which are compared against the targets z under stop-gradient using a similarity function 𝒟 (i.e., negative cosine similarity). Right: Encoder architecture (h): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.
Table 1: Ablation on main NAPE’s design elements: prediction shift, stop-gradient, and causal objective.
Pred
stop
causal
Audio Tasks
Speech Tasks
shift
grad
mask
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
✘
✓
✓
Diverge
✓
✘
✓
Diverge
✓
✓
✘
41.8
24.8
68.9
96.1
97.3
57.0
✓
✓
✓
49.6
39.1
94.2
97.9
98.8
64.9
Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.
Table 2: Ablation on the patch embedding layer.
Patch Emb.
Audio Tasks
Speech Tasks
Layer
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Convstem
46.7
34.3
89.1
97.4
98.3
63.6
Speechstem
47.6
33.1
88.4
98.1
98.9
63.0
Conv2d
49.6
39.1
94.2
97.9
98.8
64.9
Figure 3: NAPE’s performance across four scan orders on six benchmarks.
Table 3: Ablation on the predictor-style variants.
Predictor
#Par.
Audio Tasks
Speech Tasks
Style
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
None
-
48.7
37.8
93.3
98.1
98.8
64.2
2-MLP
1.2M
49.4
38.5
93.6
98.0
98.8
64.2
Transformer
14.2M
49.2
38.4
93.0
98.2
98.7
65.0
SimSiam
1.8M
49.6
39.1
94.2
97.9
98.8
64.9
Figure 4: Scaling comparison between Audio-MAE and NAPE, raster.
Table 4: Ablation on NAPE’s target to predict.
Predicted
Audio Tasks
Speech Tasks
Target
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
1st enc. layer
Diverge
Raw Mel
49.7
38.0
94.8
97.7
98.6
64.2
Patch embed
49.6
39.1
94.2
97.9
98.8
64.9
Figure 5: NAPE’s results at different scales under raster and diagonal scan orders.
Table 5: Ablation on the similarity function.
Similarity
Audio Tasks
Speech Tasks
Function
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
L1
Diverge
L2
Diverge
Cross-entropy
48.8
37.4
93.6
98.0
98.7
64.5
Cosine
49.6
39.1
94.2
97.9
98.8
64.9
Figure 6: Layer-wise linear probing analysis.
Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold, second-best are underlined.
Model
#Par.
Pre-train
Audio Tasks
Speech Tasks
Data
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Out-of-domain Supervised Pre-training
PSLA (32)
14M
IN
44.4
31.9
-
-
96.3
-
AST (31)
86M
IN
45.9
34.7
88.7
95.5
98.1
56.0
HTS-AT (17)
31M
IN
47.1
-
-
-
98.0
-
Audio-CLIP (35)
93M
TI+AS
25.9
-
96.7
-
-
-
In-domain Supervised Pre-training
AST (31)
86M
IN+AS
45.9
-
95.6
-
97.9
-
HTS-AT (17)
31M
IN+AS
47.1
-
97.0
-
-
-
Audio-MAE (40)
86M
AS
-
-
97.4
-
-
-
Self-Supervised Pre-training
SS-AST (33)
89M
AS+LS
-
31.0
88.8
96.0
98.0
59.6
MAE-AST (6)
86M
AS+LS
-
30.6
90.0
95.8
97.9
59.8
CAV-MAE (34)
86M
IN+AS
44.9
34.2
-
-
-
-
Audio-MAE (40)
86M
AS
47.3
37.1
94.1
96.9
98.3
-
Audio-MAE L (40)
304M
AS
47.4
37.7
-
-
-
-
data2vec (7)
94M
AS
-
34.5
-
-
-
-
MaskSpec (22)
86M
AS
47.1
32.3
89.6
-
97.7
-
BEATsiter3 (19)
90M
AS
48.0
38.3
95.6
97.7
98.3
64.5
A-JEPA (29)
86M
AS
48.6
38.4
96.3
97.7
98.5
-
ASiT (2)
86M
AS
48.0
38.6
95.3
98.2
98.9
-
EAT (20)
88M
AS
48.6
40.2
95.9
-
98.3
-
SSLAM (3)
88M
AS
50.2
40.9
96.2
98.8
98.1
-
SPEARa Large (68)
327M
AS
49.7
39.3
-
-
-
-
NAPE-B raster
85M
AS
49.6
39.1
94.2
97.9
98.8
64.9
NAPE-B diagonal
85M
AS
49.7
39.2
94.8
97.9
98.6
67.1
NAPE-L raster
303M
AS
50.2
40.5
96.0
97.9
98.8
68.0
NAPE-L diagonal
303M
AS
50.0
40.4
96.2
98.2
98.9
68.8
Figure 7: Top: Prediction Quality Analysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over 500 audio clips (left) and for two individual clips (middle, right). Bottom: Attention/Embedding Analyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding z^t+1 and every actual patch embedding in the spectrogram.
Table 7: Linear probing results.
Model
Layer
AS-2M
AS-20K
ESC-50
Small
2nd
23.2
18.9
79.8
Base
6th
25.0
19.7
81.7
Large
11th
27.1
20.4
83.5
Table 8: Hyperparameter list. When an hyperparameter h varies between the base (hb) and large (hl) model, we include both values like (hb/hl). ∗Following (19), we balance each class to 50% of the size of the unknown class for each training epoch.
Hyperparameters
Pre-training
Fine-tuning
AS-2M
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Optimizer
AdamW
Opt. Momentum (β1, β2)
(0.9,0.95)
(0.9,0.999)
Weight Decay
0.05
Learning Rate Scheduler
Cosine Decay
Layer-Wise LR Decay
1.0
0.7/0.9
0.8/0.9
0.7/0.9
0.7/0.8
0.7/0.8
0.7/0.9
Base Learning Rate
5e-3
1.25e-3
Epochs
30/25
20/15
30/20
100
50
50
50
Warm-up Epochs
3
4/3
6/5
10
5
5
5
Batch Size
256/128
64
GPUs
8
4
4
1
1
1
4
Weighted sampling
✗
✓
✗
✗
✓∗
✗
✗
Multilabel
N/A
✓
✓
✗
✗
✗
✗
EMA Decay Rate
0.9999
0.99995
0.999
✗
✗
✗
✗
Label Smoothing
N/A
0.
0.
0.1
0.1
0.
0.1
Roll Augmentation
✗
✓
✓
✓
✗
✗
✓
Drop Path
0.
0.1
SpecAug (time/freq)
N/A
(96,16)
(24,16)/(96,16)
(96,24)/(24,16)
(24,16)
(24,16)
(48,24)
Mixup (alpha/prob.)
N/A
(0.8,1.0)
(0.8,0.8)
(0.8,0.5)
(0.8,0.8)
(0.8,0.8)
(0.8,0.5)
Cutmix (alpha/prob.)
N/A
(1.0,1.0)
(1.0,0.8)
(1.0,0.5)
(1.0,0.8)
(1.0,0.8)
(1.0,0.5)
Noise Augmentation
✗
✓
✓
✓
✓
✓
✓
Loss Function
Neg Cos Sim
BCE
BCE
CE
BCE
BCE
CE
Dataset Mean for Norm.
-6.84
-6.84
-6.84
-6.84
-9.11
-9.16
-13.74
Dataset Std for Norm.
5.38
5.38
5.38
5.38
4.53
4.61
3.88
Table 9: Ablation on the pooling method and attention type at fine-tuning.
Attention
Pooling
Task
Type
Mode
AS-2M
AS-20K
KS2
Bidirec
CLS Tok
49.7
38.9
98.7
Bidirec
Last Tok
49.6
38.7
98.8
Bidirec
Avg Pool
49.6
39.1
98.8
Causal
Last Tok
49.4
38.9
98.8
Table 10: Ablation on freezing the emb. layer.
Freeze emb
AS-2M
AS-20K
KS2
✓
49.61
39.14
98.73
✘
49.58
39.08
98.80
Figure 8: Pre-training budget ablation. AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.
Table 12: Ablation on normalization layer style.
Norm
Audio Tasks
Speech Tasks
Type
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
LN
49.6
39.1
94.2
97.9
98.8
64.9
RMS
49.5
38.8
94.4
97.8
98.8
65.2
Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures.
Table 13: Ablation on random masking applied to the input embeddings.
Mask
Audio Tasks
Speech Tasks
Ratio
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
0
47.6
36.2
92.9
97.4
98.3
64.6
20
47.3
36.2
92.2
97.6
98.5
63.9
50
47.2
35.2
91.7
97.5
98.3
63.9
왜 중요한가
언어모델에서 통했던 '다음 것 예측하기'라는 단순한 학습 방식이 소리에도 그대로 통한다는 것을 보여주므로, 복잡한 사전학습 레시피 없이도 오디오 AI 모델을 더 쉽게 만들 수 있는 길을 연다. 특히 적은 학습 횟수로도 기존 방법들과 맞먹는 성능을 낼 수 있어 계산 자원이 부족한 환경에서도 매력적인 선택지가 된다.
이 논문의 용어
자기지도학습(Self-supervised learning) · 사람이 붙인 정답 라벨 없이 데이터 자체의 구조로부터 학습하는 방식
로그멜 스펙트로그램 · 소리를 시간과 주파수 축으로 표현한 그림 형태의 데이터
인과적 마스킹 · 모델이 미래 시점의 정보를 미리 보지 못하게 가리는 장치
스톱그래디언트 · 특정 쪽 경로에는 학습 신호(그래디언트)가 흐르지 않도록 막는 기법
선형 프로빙 · 사전학습된 모델은 그대로 두고 그 위에 얹은 간단한 분류기만 학습시켜 성능을 확인하는 평가 방법
논문 원문 초록 (영문)
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
