Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
arXiv:2608.198632026-08-19
Training an audio model to just guess the next spectrogram patch is enough to reach top performance
NAPE is a method that trains a model by showing it slices of a sound's time-frequency picture in order and asking it to predict the feature values of the next slice. It skips heavy machinery like reconstruction decoders or teacher-student setups, relying only on a causal mask and a gradient-stopping trick to make learning stable. Across six audio and speech benchmarks, it reached state-of-the-art fine-tuning results on several tasks and kept improving as the model got bigger.
What they did
Sound is converted into a log-mel spectrogram (a time-vs-frequency picture of audio), cut into small square patches, ordered in sequence, and a Transformer is trained to predict the next patch's embedding using only the patches that came before it
Common tricks from prior audio self-supervised methods—reconstruction decoders, separate acoustic tokenizers, EMA teacher-student setups, extra regularization losses—are all removed; only a causal attention mask (hides future patches) and stop-gradient (blocks learning signal from the target side) are used
Four ways of ordering the 2D patch grid into a 1D sequence were compared (raster, time-major, zigzag, diagonal), and the three orders that progress through time (raster, zigzag, diagonal) clearly beat time-major, which exhausts frequency first
On six benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP), NAPE achieved state-of-the-art fine-tuning results on several tasks and scaled consistently across three model sizes (19M, 85M, 303M parameters)
Even with the encoder frozen and only a classifier trained on top (linear probing), NAPE performed strongly, and its attention patterns showed structured, meaningful behavior without any explicit supervision
Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddings z are processed by the causal encoder h and predictor g to produce predictions z^, which are compared against the targets z under stop-gradient using a similarity function 𝒟 (i.e., negative cosine similarity). Right: Encoder architecture (h): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.
Table 1: Ablation on main NAPE’s design elements: prediction shift, stop-gradient, and causal objective.
Pred
stop
causal
Audio Tasks
Speech Tasks
shift
grad
mask
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
✘
✓
✓
Diverge
✓
✘
✓
Diverge
✓
✓
✘
41.8
24.8
68.9
96.1
97.3
57.0
✓
✓
✓
49.6
39.1
94.2
97.9
98.8
64.9
Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.
Table 2: Ablation on the patch embedding layer.
Patch Emb.
Audio Tasks
Speech Tasks
Layer
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Convstem
46.7
34.3
89.1
97.4
98.3
63.6
Speechstem
47.6
33.1
88.4
98.1
98.9
63.0
Conv2d
49.6
39.1
94.2
97.9
98.8
64.9
Figure 3: NAPE’s performance across four scan orders on six benchmarks.
Table 3: Ablation on the predictor-style variants.
Predictor
#Par.
Audio Tasks
Speech Tasks
Style
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
None
-
48.7
37.8
93.3
98.1
98.8
64.2
2-MLP
1.2M
49.4
38.5
93.6
98.0
98.8
64.2
Transformer
14.2M
49.2
38.4
93.0
98.2
98.7
65.0
SimSiam
1.8M
49.6
39.1
94.2
97.9
98.8
64.9
Figure 4: Scaling comparison between Audio-MAE and NAPE, raster.
Table 4: Ablation on NAPE’s target to predict.
Predicted
Audio Tasks
Speech Tasks
Target
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
1st enc. layer
Diverge
Raw Mel
49.7
38.0
94.8
97.7
98.6
64.2
Patch embed
49.6
39.1
94.2
97.9
98.8
64.9
Figure 5: NAPE’s results at different scales under raster and diagonal scan orders.
Table 5: Ablation on the similarity function.
Similarity
Audio Tasks
Speech Tasks
Function
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
L1
Diverge
L2
Diverge
Cross-entropy
48.8
37.4
93.6
98.0
98.7
64.5
Cosine
49.6
39.1
94.2
97.9
98.8
64.9
Figure 6: Layer-wise linear probing analysis.
Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold, second-best are underlined.
Model
#Par.
Pre-train
Audio Tasks
Speech Tasks
Data
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Out-of-domain Supervised Pre-training
PSLA (32)
14M
IN
44.4
31.9
-
-
96.3
-
AST (31)
86M
IN
45.9
34.7
88.7
95.5
98.1
56.0
HTS-AT (17)
31M
IN
47.1
-
-
-
98.0
-
Audio-CLIP (35)
93M
TI+AS
25.9
-
96.7
-
-
-
In-domain Supervised Pre-training
AST (31)
86M
IN+AS
45.9
-
95.6
-
97.9
-
HTS-AT (17)
31M
IN+AS
47.1
-
97.0
-
-
-
Audio-MAE (40)
86M
AS
-
-
97.4
-
-
-
Self-Supervised Pre-training
SS-AST (33)
89M
AS+LS
-
31.0
88.8
96.0
98.0
59.6
MAE-AST (6)
86M
AS+LS
-
30.6
90.0
95.8
97.9
59.8
CAV-MAE (34)
86M
IN+AS
44.9
34.2
-
-
-
-
Audio-MAE (40)
86M
AS
47.3
37.1
94.1
96.9
98.3
-
Audio-MAE L (40)
304M
AS
47.4
37.7
-
-
-
-
data2vec (7)
94M
AS
-
34.5
-
-
-
-
MaskSpec (22)
86M
AS
47.1
32.3
89.6
-
97.7
-
BEATsiter3 (19)
90M
AS
48.0
38.3
95.6
97.7
98.3
64.5
A-JEPA (29)
86M
AS
48.6
38.4
96.3
97.7
98.5
-
ASiT (2)
86M
AS
48.0
38.6
95.3
98.2
98.9
-
EAT (20)
88M
AS
48.6
40.2
95.9
-
98.3
-
SSLAM (3)
88M
AS
50.2
40.9
96.2
98.8
98.1
-
SPEARa Large (68)
327M
AS
49.7
39.3
-
-
-
-
NAPE-B raster
85M
AS
49.6
39.1
94.2
97.9
98.8
64.9
NAPE-B diagonal
85M
AS
49.7
39.2
94.8
97.9
98.6
67.1
NAPE-L raster
303M
AS
50.2
40.5
96.0
97.9
98.8
68.0
NAPE-L diagonal
303M
AS
50.0
40.4
96.2
98.2
98.9
68.8
Figure 7: Top: Prediction Quality Analysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over 500 audio clips (left) and for two individual clips (middle, right). Bottom: Attention/Embedding Analyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding z^t+1 and every actual patch embedding in the spectrogram.
Table 7: Linear probing results.
Model
Layer
AS-2M
AS-20K
ESC-50
Small
2nd
23.2
18.9
79.8
Base
6th
25.0
19.7
81.7
Large
11th
27.1
20.4
83.5
Table 8: Hyperparameter list. When an hyperparameter h varies between the base (hb) and large (hl) model, we include both values like (hb/hl). ∗Following (19), we balance each class to 50% of the size of the unknown class for each training epoch.
Hyperparameters
Pre-training
Fine-tuning
AS-2M
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Optimizer
AdamW
Opt. Momentum (β1, β2)
(0.9,0.95)
(0.9,0.999)
Weight Decay
0.05
Learning Rate Scheduler
Cosine Decay
Layer-Wise LR Decay
1.0
0.7/0.9
0.8/0.9
0.7/0.9
0.7/0.8
0.7/0.8
0.7/0.9
Base Learning Rate
5e-3
1.25e-3
Epochs
30/25
20/15
30/20
100
50
50
50
Warm-up Epochs
3
4/3
6/5
10
5
5
5
Batch Size
256/128
64
GPUs
8
4
4
1
1
1
4
Weighted sampling
✗
✓
✗
✗
✓∗
✗
✗
Multilabel
N/A
✓
✓
✗
✗
✗
✗
EMA Decay Rate
0.9999
0.99995
0.999
✗
✗
✗
✗
Label Smoothing
N/A
0.
0.
0.1
0.1
0.
0.1
Roll Augmentation
✗
✓
✓
✓
✗
✗
✓
Drop Path
0.
0.1
SpecAug (time/freq)
N/A
(96,16)
(24,16)/(96,16)
(96,24)/(24,16)
(24,16)
(24,16)
(48,24)
Mixup (alpha/prob.)
N/A
(0.8,1.0)
(0.8,0.8)
(0.8,0.5)
(0.8,0.8)
(0.8,0.8)
(0.8,0.5)
Cutmix (alpha/prob.)
N/A
(1.0,1.0)
(1.0,0.8)
(1.0,0.5)
(1.0,0.8)
(1.0,0.8)
(1.0,0.5)
Noise Augmentation
✗
✓
✓
✓
✓
✓
✓
Loss Function
Neg Cos Sim
BCE
BCE
CE
BCE
BCE
CE
Dataset Mean for Norm.
-6.84
-6.84
-6.84
-6.84
-9.11
-9.16
-13.74
Dataset Std for Norm.
5.38
5.38
5.38
5.38
4.53
4.61
3.88
Table 9: Ablation on the pooling method and attention type at fine-tuning.
Attention
Pooling
Task
Type
Mode
AS-2M
AS-20K
KS2
Bidirec
CLS Tok
49.7
38.9
98.7
Bidirec
Last Tok
49.6
38.7
98.8
Bidirec
Avg Pool
49.6
39.1
98.8
Causal
Last Tok
49.4
38.9
98.8
Table 10: Ablation on freezing the emb. layer.
Freeze emb
AS-2M
AS-20K
KS2
✓
49.61
39.14
98.73
✘
49.58
39.08
98.80
Figure 8: Pre-training budget ablation. AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.
Table 12: Ablation on normalization layer style.
Norm
Audio Tasks
Speech Tasks
Type
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
LN
49.6
39.1
94.2
97.9
98.8
64.9
RMS
49.5
38.8
94.4
97.8
98.8
65.2
Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures.
Table 13: Ablation on random masking applied to the input embeddings.
Mask
Audio Tasks
Speech Tasks
Ratio
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
0
47.6
36.2
92.9
97.4
98.3
64.6
20
47.3
36.2
92.2
97.6
98.5
63.9
50
47.2
35.2
91.7
97.5
98.3
63.9
Why it matters
It shows that the simple 'predict the next thing' recipe that worked for language models also works for audio, opening a path to building strong audio AI without elaborate pre-training pipelines. It also matches or beats many published baselines after only a few epochs of pre-training, making it attractive under tight compute budgets.
Terms in this paper
Self-supervised learning · Training a model from the structure of the data itself, without human-annotated labels
Log-mel spectrogram · A picture-like representation of sound showing frequency content over time
Causal attention mask · A mechanism that prevents the model from seeing future positions in the sequence
Stop-gradient · A technique that blocks the learning signal from flowing back through one specific branch
Linear probing · An evaluation where the pre-trained model is frozen and only a simple classifier on top is trained
Original abstract (English)
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
