Figure 1: Interpretable front-end made as a collection of branches, the k-th branch is highlighted in blue. During training each branch gets matched to a neural source with specific spatial and dynamical properties.Figure 2: Our network’s architecture for MEG-to-audio embedding alignment. A 3-second, 208-channel MEG segment is processed by an interpretable front end: spherical-harmonic 3D spatial attention maps the sensor signals to 270 geometry-constrained virtual channels; a 1×1 unmixing convolution applies a learned affine transformation in this channel space; the subject-specific layer then projects the representation to K interpretable branches selected by the subject ID. Each branch is passed through a depthwise temporal convolution with a 150 ms kernel, producing filtered branch-wise signals. These signals are processed by a convolutional module with B residual convolutional blocks, where we evaluate B∈{0,…,5} and use B=2 in the main architecture, followed by a convolutional head that outputs the MEG embedding aligned with the wav2vec audio embedding.
Table 1: Trainable model size and retrieval performance for representative configurations. F denotes the dimensionality of the target audio representation. The K=270, five-block LISA model is the closest tested configuration to Défossez et al. [15] in branch count and decoder depth, but is not a reproduction: it retains our 3D attention, temporal filters, preprocessing, and training procedure. The published Défossez et al. scores use a different preprocessing and evaluation protocol (word-aligned test segments and without ocular or cardiac component removal) and are included only to ground the external parameter comparison.
Model
K
B
F
Parameters
Candidates
Top-1 (%)
Top-10 (%)
LISA, smaller branch space
15
2
768
380,769
1005
39.51
70.23
LISA, LinearDR-12
25
2
12
427,651
1005
39.95
70.54
LISA, no convolutional blocks
25
0
768
471,219
1005
36.76
67.37
LISA, main
25
2
768
486,619
1005
40.01
70.60
LISA, five convolutional blocks
25
5
768
509,719
1005
39.06
69.82
LISA, closest tested capacity
270
5
768
7,210,224
1005
36.41
67.46
Défossez et al. [15]
270
5
1024
9,565,054
1363
41.30
70.70
Figure 3: Retrieval accuracy as a function of the number of interpretable branches K and the number of convolutional blocks in the decoder. For each configuration, Top-1 and Top-10 test accuracy are reported at the checkpoint with the lowest validation loss. Accuracy increases sharply from very small K to approximately K=10–25, then enters a broad plateau; larger K does not produce systematic gains and can mildly degrade performance. Across decoder depths, the 0-block model is consistently weaker, while models with 2–5 convolutional blocks form a similar high-performing regime. The main 2-conv, K=25 configuration lies on this compact high-accuracy plateau.Figure 4: 3D spherical-harmonic attention learned by architectures with varying numbers of non-linear convolutional blocks (B) and branch counts (K=5,10,25). Neff=(∑mpm2)−1 is the inverse-Simpson effective number of sensors, where pm=c¯m/∑m′c¯m′; smaller values indicate that attention is concentrated on fewer sensors. For visualization maximal value was capped to 99-th percentile, but all Neff are computed with full attention weights without clipping.
Figure 5: Leading ten singular vectors of the across-subject spatial filter and spatial pattern matrices aggregated from the interpretable branches of all 27 subjects, and the equivalent current dipoles fitted to the subspace spanned by these topographies. We can clearly observe the involvement of not only the primary auditory cortices but also frontal and medial temporal lobe structures.(b) Dominant dipoles derived from the cross-subject spatial patterns of Figure 5(a) using the RAP-MUSIC algorithm [51] (subspace correlation threshold 0.8). Dipoles cluster in bilateral auditory cortices, medial temporal lobe and frontal lobe on the fsaverage anatomy.
Figure 6: The 12 largest clusters for the main K=25 model, computed using the Cable Spool Fort recordings from the first session. Each column shows the medoid of one cluster. Rows show, from top to bottom, the sensor-space spatial pattern, the temporal pattern computed using the zero-mean temporal kernel, its magnitude spectrum, and the corresponding MNE-Python [30] source-magnitude estimate on the fsaverage surface in left- and right-hemisphere lateral views.Figure 7: Paired MEG occlusion effects for 19 stimulus features. For each participant, the plotted effect is the retrieval-rank difference between feature-absent replacement (“removal”) and matched feature-present replacement (“control”), after averaging donor realisations, eligible queries, and multiple sessions. Positive values indicate that preserving feature-associated MEG information retained a better rank. Grey points show participant effects and diamonds show group means. Green violins show the feature-wise sign-flip null distributions in rank-difference units; stars mark one-sided single-step max-T familywise-error-corrected p<0.05. Every rank was computed against the complete, unchanged bank of 1005 candidates. Because the masks differed in duration and in their sets of eligible queries, effect magnitudes should not be read as a calibrated ranking of feature-encoding strength across features.
Figure 8: Test retrieval accuracy as a function of paired MEG–audio segment duration for models with K=25 branches and B∈{0,2,5} convolutional blocks. Each point reports final-test accuracy from the checkpoint with the lowest validation loss. Audio embeddings were regenerated directly from the continuous sounds at every duration, and all conditions used the same 5 s-feasible anchor set and the same 991-candidate retrieval database. The 3 s points belong to this regenerated ablation and are distinct from the main 1005-candidate evaluation.Figure 9: Top-n test retrieval accuracy for the main two-block, K=25 model across paired MEG–audio segment durations. Curves were calculated using the similarity rank of the correct audio segment among the same 991 candidates. The advantage of longer segments is present throughout the evaluated range n=1,…,50.
Figure 10: Feature-space compression of the target wav2vec representation. The feature dimension is reduced using either a fixed PCA projection or a trainable linear projection optimized with the retrieval loss. Learned feature reduction preserves retrieval accuracy over a wide range of dimensions, whereas PCA degrades substantially faster in the low-dimensional regime.Figure 11: Temporal-resolution compression of the target wav2vec trajectory. Temporal PCA, trainable linear reduction, local pooling, and global pooling methods are compared. Unlike feature compression, temporal compression causes a clear performance loss, and global pooling collapses to near-chance retrieval.
Short segments of perceived speech can be retrieved from non-invasive magnetoencephalographic (MEG) recordings by deep networks trained with a CLIP-style objective against wav2vec 2.0 audio embeddings. Yet their weights do not map onto electrophysiological quantities, and it remains unclear which speech properties drive retrieval. We build on a high-performing MEG-to-audio retrieval architecture but redesign both its front end and decoder. Its spatial attention operates on a flattened sensor lay
![(b) Dominant dipoles derived from the cross-subject spatial patterns of Figure 5(a) using the RAP-MUSIC algorithm [51] (subspace correlation threshold 0.8). Dipoles cluster in bilateral auditory cortices, medial temporal lobe and frontal lobe on the fsaverage anatomy.](https://media.metallab.ai/papers/2608.01481/f5.svg)
![Figure 6: The 12 largest clusters for the main K=25 model, computed using the Cable Spool Fort recordings from the first session. Each column shows the medoid of one cluster. Rows show, from top to bottom, the sensor-space spatial pattern, the temporal pattern computed using the zero-mean temporal kernel, its magnitude spectrum, and the corresponding MNE-Python [30] source-magnitude estimate on the fsaverage surface in left- and right-hemisphere lateral views.](https://media.metallab.ai/papers/2608.01481/f6.png)
