AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox›
The Loss Does Not See the Basis, but Adam Does
arXiv:2608.051362026-08-04
Gradient descent and Adam quietly pick different matrices to converge to — because of a symmetry the loss can't see
When a model's weight matrix is written as a product W=UV^T, gradient descent tends to settle on a simple, low-rank solution, but Adam, started from the exact same small initialization, does not. The authors trace this to a 'gauge symmetry' — rotating U and V by the same orthogonal matrix leaves the loss unchanged — and classify nine common optimizers by whether their update rule respects this symmetry. That classification predicts, and is confirmed by, actual performance differences on matrix recovery, transformer attention, and real hyperspectral image datasets.
METAL LAB explanatory visual
How gauge symmetry splits optimizers into two classes
Evidence statusMeasured results reported
Gauge symmetryRotating U and V in W=UV^T by the same orthogonal matrix Q leaves W and the loss unchanged
Equivariant optimizersGD, momentum, shared-scalar Adam, Muon, Shampoo — trace the same trajectory regardless of the rotated starting basis
Coordinate-wise optimizersAdam, RMSProp, signSGD, Lion — process each coordinate separately, so their trajectory depends on which basis they start in
Matrix recovery & attention testsThe two classes show a clean separation in recovery error and in how far apart attention-head internals end up
Real-data checkOn hyperspectral image completion, the equivariant method (GD) achieves lower held-out error than Adam at matched training loss
An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.
What they did
A factored model W=UV^T is unchanged if you rotate both U and V by the same orthogonal matrix Q (U to UQ, V to VQ); this invariance of the loss is called gauge symmetry.
Sorting optimizers by whether their update respects this symmetry ('gauge-equivariance') places gradient descent, momentum, a 'shared-scalar' variant of Adam, Muon, and Shampoo on one side, and standard Adam, RMSProp, and similar coordinate-wise methods on the other.
In a controlled matrix-sensing test (recovering a hidden rank-3, 40x40 matrix from underdetermined measurements), all nine optimizers were run until training loss vanished, then compared by recovery error: equivariant methods reached errors of at most 0.286, while coordinate-wise methods stayed above 0.42, with no overlap.
A tunable 'dial' experiment that smoothly interpolates between Adam's standard per-coordinate denominator and a single shared scalar showed that recovery quality and simplicity of the solution degrade monotonically as the update becomes more coordinate-specific, isolating this anisotropy as the actual cause.
In a small transformer, two copies computing the identical function but written in different (rotated) coordinate bases were trained with different optimizers: Adam split the two copies apart from the very first update step and ended with attention-head internal values (WQ^T WK) differing by 56%, while equivariant optimizers stayed within floating-point noise of each other throughout.
Figure 1: Equivariance structures the set of available optimizers under a fixed computational budget. Ground-truth recovery for nine optimizer rules (eight standard rules plus a shared-scalar Adam as a baseline) on an underdetermined matrix sensing task without weight decay. All are run until the training residuals vanish (interpolation), which rules out any performance difference due to residual training error. The four gauge-equivariant methods (blue) maintain the low-rank bias while the five coordinate-wise approaches (vermilion) do not: on this task and at this budget, recovery error is at most 0.286 for every equivariant method and above 0.42 for every coordinate-wise one, leaving the space between the two unoccupied. That qualifier matters, as one of the coordinate-wise methods eventually reaches similar performance beyond the 2×104-step budget used here (Appendix D.10). Muon’s label reads 6.8×10−6, the unrounded value of the 0.0000 entry in Table 2.
Table 1: Closest prior and concurrent work. The recurring distinction is whether an update respects a relevant orthogonal symmetry; we ask what it predicts for interpolant selection in factored models at matched training loss.
work
setting and result
relation to this paper
VectorAdam; Xie et al. 2025
Examine ambient rotations of vector parameters under general loss functions. Show Adam is sensitive to such rotations, whereas shared scalar updates restore equivariance.
We focus instead on an internal factorization gauge and connect the same principle to solution selection among interpolants, rather than optimization speed.
LoRA-RITE; DePavia et al. 2025
Study transformations of LoRA factors and rotations in data or feature space. Demonstrate improved optimization through invariance and altered implicit bias in Adam under rotations.
We isolate the intrinsic gauge of the factorization itself and evaluate recovery performance at matched training loss across a range of optimizers.
Silverstein et al. 2026; Zhang 2026
Analyze per-head query–key (QK) rotations and Schur multiplicity bases. Highlight that optimizer design can intentionally break or preserve symmetry.
We maintain the gauge structure intact and use it to analyze solutions chosen by standard, deployed optimizers.
Lau & Su 2026; Shirodkar 2026
Investigate architectural symmetry groups and construct optimizers that exactly respect equivariance.
These works support symmetry-aware design; we instead assess the practical consequences of preserving or violating such symmetry.
Kang et al. 2026; Dragutinović et al. 2026
Study Muon’s spectral dynamics and its impact on simplicity bias.
We extend this by mapping conditions under which uniform spectral growth helps or harms performance as the target develops a spectral tail.
Figure 2: Adam is basis-dependent in attention, shown here by the relative logit distance between twins in the gauge and noise experiments. In the former the two copies are initialized as the same function, while in the latter they differ by a small (10−7) perturbation in the same basis. Adam’s gauge twins (solid vermilion line) split structurally in one step, to a value four orders of magnitude greater than its own noise twin (dotted line), and then saturate. The same is not observed for SGD (heavy-ball momentum 0.9) or scalar-Adam, whose gauge twins remain at float noise. For Muon the gauge and noise twins trace the same curve, indicating that its separation is numerical chaos stemming from msign, not basis dependence.
Table 2: The optimizer-zoo map. Matrix sensing, 40×40, rank 3 ground truth, m=2×dof, no weight decay, three paired random seeds, all nine optimizer runs continued until interpolation (final training loss reported). Recovery is measured as ‖W−X∗‖F/‖X∗‖F (lower is better); erank is effective rank; bal is ‖U⊤U−V⊤V‖F; the notes column names the theoretical framework each row falls under—Gunasekar et al. 2017 for gradient descent (GD) and Wilson et al. 2017 for Adam. (cosine) marks the four methods that need cosine annealing to interpolate; the first row is a convex baseline, not an optimizer, recovering the minimum-nuclear-norm solution of the same measurements (both are unpacked in the text above). All values are means over the three seeds, with no dispersion quoted.
method
recovery ↓
erank
bal
train loss
notes
min-nuclear-norm
0.0335
3.30
—
—
convex reference, not an optimizer
equivariant
Muon (cosine)
0.0000
2.95
1.65
5.7×10−8
near-exact (6.8×10−6 unrounded)
GD
0.1312
4.51
0.06
6.5×10−8
Gunasekar anchor
scalar-Adam (p=0)
0.2010
5.43
0.14
3.5×10−8
equivariant; flow proxy
Shampoo
0.2856
6.95
1.47
4.5×10−8
coord.-wise
Lion (cosine)
0.4248
10.58
7.35
8.0×10−8
signum (cosine)
0.4454
7.83
1.79
6.5×10−8
low-rank-but-wrong
RMSProp (cosine)
0.5266
12.80
4.74
6.0×10−8
needs decay; Appendix D.7
Adafactor
0.5430
10.72
3.40
5.6×10−8
factored diag. still breaks
Adam
0.5734
14.37
5.37
1.2×10−11
Wilson anchor
Figure 4: The spectral-tail phase diagram. Recovery vs. the target’s tail-amplitude parameter τ (the tail carries τ2 of the target’s energy). Adam has the worst recovery error at every τ plotted. Within the equivariant class, Muon’s aggressive equal-rate schedule is exact at τ=0 and crosses over to GD near τ∗≈0.2. At the largest τ, all methods are near the measurement-determined floor set by the unidentifiable tail, and GD and Shampoo are within seed noise, so the ordering there is uninformative. Bands show ±1 s.d. over 10 seeds. The boundary is unchanged when every method is given the identical cosine schedule (Appendix D).
Table 3: Adam is basis-dependent in attention, and the noise twin separates structure from chaos. The values are relative logit distances between twins on validation inputs for the same task; multi-seed statistics are shown in Appendix D.5. At the harness’s default CPU precision, the baseline rounding threshold is the step-0 column (≈2×10−7, i.e. one function evaluated across two bases); the equivariant methods reach this floor at step 1, and their later nonzero entries result from numerical noise, not from gauge-symmetry violations—Muon’s split in particular, as per Proposition 4.2. These discrepancies are reduced, if not eliminated, in GPU float64, where the step-1 values fall within 2.8×10−16–1.1×10−15 over the three methods and four configurations; Table 9 carries the worst case in each.
optimizer
twin type
step 0
step 1
step 100
step 1500 (final)
Adam
gauge
1.8×10−7
3.6×𝟏𝟎−𝟑
6.5×10−1
7.7×10−1
Adam
A=I
0
0
0
0
Adam
noise (10−7)
2.6×10−7
2.9×10−7
1.7×10−5
1.6×10−5
SGD
gauge
1.8×10−7
2.9×10−7
3.7×10−5
2.0×10−5
scalar-Adam
gauge
1.8×10−7
1.6×10−7
4.1×10−6
4.3×10−6
Muon
gauge
1.8×10−7
2.2×10−7
2.3×10−2
8.5×10−1
Muon
noise (10−7)
2.6×10−7
2.9×10−7
4.0×10−2
8.7×10−1
Figure 5: Performance as a function of training loss on real data (Indian Pines, m/dof≈1.15, 4 random seeds, GPU float64; learning rates chosen via the train-only rule in Appendix D.6). This is the corresponding row of Table 5, read along its whole path rather than at its endpoint alone. Gradient descent (GD) sits below Adam at every matched training loss beyond the shared starting point, and the two are compared only at equal training loss, so no fitted run is ever set against an unfitted one. Left panel: held-out error; bands are ±1 s.d. over the 4 seeds, hidden inside the line width wherever the seeds agree—the whole path for GD—and opening up only where they do not, as Adam’s do once overfitting begins. Right panel: effective rank. GD settles between 5.6 and 13.2; Adam climbs to 28, against the scene’s intrinsic rank of 24; Muon tracks the rank cap until convergence, then drops sharply.
Table 4: Spectral-tail phase diagram (recovery; 10 seeds, float64). Muon is exact at τ=0, degrades fastest as tail energy grows, and cedes to GD in a crossover region near τ∗≈0.2 (≈4% tail energy). Bold marks each row’s best recovery, with both members of a within-noise tie bolded; the seed dispersions behind those ties are the ±1 s.d. bands of Figure 4, omitted here for space. Decay-symmetrized rows in Appendix D.
τ
GD
Adam
Muon
Shampoo
regime
0.00
0.112
0.542
0.000
0.334
Muon exact
0.05
0.150
0.543
0.095
0.343
Muon
0.10
0.214
0.555
0.191
0.375
Muon
0.20
0.351
0.597
0.354
0.449
boundary (≈4% tail); Muon cedes
0.35
0.549
0.685
0.580
0.576
GD
0.50
0.727
0.776
0.752
0.723
tail regime; GD/Shampoo within noise
Table 5: Matched-loss recovery on two hyperspectral datasets (held-out RMSE ×10−2, mean ± standard deviation over 4 random seeds, evaluated at matched training loss ≤10−5, GPU float64; learning rates as in Appendix D.6). The reduction column reports the percent improvement of GD over Adam, as (Adam−GD)/Adam using mean performance over seeds; gradient descent (GD) outperforms Adam in every individual seed of every entry. In parentheses, Muon’s effective rank at the deepest point in the fit (after the collapse typical of the late stages), given a model capacity of 48.
dataset
m/dof24
GD
Adam
RMSE reduction
Muon (rank)
Indian Pines
1.15
1.481±0.013
2.600±0.047
+43.0%
3.397±0.082 (36)
1.9
1.244±0.012
1.718±0.035
+27.6%
3.210±0.122 (35)
Pavia Univ.
1.15
0.819±0.026
1.458±0.123
+43.8%
1.260±0.121 (10)
1.9
0.615±0.010
0.798±0.023
+22.9%
0.770±0.017 0(8)
Table 6: Recovery under three schedules (3 seeds, best-recovery lr per cell) for five reference methods: the equivariant anchors GD, scalar-Adam, and Shampoo, the coordinate-wise anchor Adam, and the constant-norm case Muon. The split does not move; under a single uniform cosine schedule the complete nine-method classification is again 9/9 (selection-rule test, Appendix D.3).
method
constant
cosine
cosine, full horizon
GD
0.1312
0.1312
0.1312
Adam
0.5734
0.5734
0.5764
scalar-Adam
0.2010
0.2010
0.2010
Muon
(0.0084†)
0.0000
0.0000
Shampoo
0.2856
0.2856
0.2856
Table 7: Recovery versus learning rate for the five reference methods, averaged over three random seeds. For each method the results occupy two rows: the top row gives the learning-rate grid used (of varying length, since Adam has five rates while the rest have four), and the second row the corresponding recovery, so a method’s row simply ends where its grid does. “→fl” signifies the lowest learning rate at which interpolation still succeeds, the flow-limit-aligned column; n/i = does not interpolate; div = diverges.
method
learning-rate grid
GD
lr
0.003→fl
0.01
0.03
0.1
recovery
0.113
0.131
0.437
div
Adam
lr
0.0003→fl
0.001
0.003
0.01
0.03
recovery
0.560
0.576
0.570
0.573
0.581
scalar-Adam
lr
0.0003→fl
0.001
0.003
0.01
recovery
0.165
0.201
0.256
0.357
Muon (cosine)
lr
0.003
0.01
0.03
0.1
recovery
0.915
0.748
0.000
0.000
Shampoo
lr
0.01
0.03
0.1
0.3
recovery
(0.881 n/i)
0.286
div
div
Table 8: Recovery vs. initialization scale (3 seeds). The split holds at 10−3 (the paper’s setting) and 3×10−3; at 10−2 the small-init bias fades for every method except Muon, which remains exact, marking its bias as schedule- rather than init-driven.
init
GD
Adam
scalar-Adam
Muon
Shampoo
10−3
0.131
0.573
0.201
0.000
0.286
3×10−3
0.203
0.575
0.260
0.000
0.352
10−2
0.318
0.584
0.358
0.000
0.575
Table 9: Twin drift measured at scale and on real text, in GPU float64 precision. For each configuration, Adam runs 6 init×draw gauge pairs while each equivariant method runs 2; the A=I determinism twin is evaluated for Adam at every configuration and is exactly 0 throughout. “eq. step 1” is the maximum (worst) gauge drift across the three equivariant methods (SGD, scalar-Adam, Muon) at step 1, all at machine precision; “ratio” is the onset ratio defined above. The character-level language model twins use the same deterministic minibatch stream and reach comparable validation losses (1.579 vs. 1.585).
config
Adam gauge, step 1
eq., step 1
Adam, final
ratio (step 1)
mod-47, 2L, d=64
4.0–5.7×10−3
5.4×10−16
0.69–0.78
93×
mod-97, 4L, d=128
8.2–11×10−3
7.3×10−16
0.66–0.75
158×
mod-97, 6L, d=256
6.5–7.2×10−3
1.1×10−15
0.61–0.64
94×
text (char-LM), 6L, d=256
2.2–2.4×10−3
8.0×10−16
0.36–0.37
39×
Table 10: FlowAdam-p=0 recovery along a trajectory aligned by training loss (extended 1.2×105-step budget, 3 seeds). The mean at the 10−7 threshold (0.1691) matches the original 3×104-step result exactly and improves upon the dial-only baseline (0.2010) by +15.9%. Its effective rank (4.8–5.4) is the closest of any Adam variant in this study to gradient descent’s (4.51).
train loss ≤
10−4
10−5
3×10−6
10−6
10−7
erank
seed 42
0.1404
0.1401
0.1401
0.1401
0.1401
4.77
seed 123
0.1558
0.1556
0.1555
0.1555
0.1555
4.84
seed 456
0.2120
0.2117
0.2117
0.2117
0.2117
5.41
mean
0.1694
0.1691
0.1691
0.1691
0.1691
5.01
Table 11: Optimization algorithms on the Zoo recovery task, across problem sizes (10-seed averages; float64, H100 GPU, 4×104 steps). The ranking is consistent across scales: gradient descent (GD), scalar-Adam (p=0), and Muon significantly outperform coordinate-wise Adam, RMSProp, Lion, and Adafactor, with Adam-type methods at 1.7–4.3× GD’s error at every size and Muon almost exact (error <10−5 for all seeds) up to n=256. The errors of GD and scalar-Adam are limited by the step-size choice (Remark A.5), so the relative ordering, not the absolute value, is the conclusion.
equivariant
coordinate-wise
n
GD
sc-Adam
Muon
Shampoo†
Adam
RMSProp
Lion
Adafactor
signum‡
64
0.143
0.195
0.000
0.646
0.574
0.596
0.483
0.612
0.000
96
0.202
0.279
0.000
0.595
0.609
0.615§
0.548
0.687
0.000
128
0.248
0.345
0.000
0.659
0.643
0.617
0.602
0.745
0.000
192
0.346
0.468
0.000
0.838
0.696
0.669
0.760
0.817
0.000
256
0.410
0.548
0.000
0.918
0.741
0.708
0.895
0.867
0.000
Findings
On the matrix-sensing recovery task, gauge-equivariant optimizers (GD, shared-scalar Adam, Muon, Shampoo) reached recovery error at most 0.286, while coordinate-wise optimizers (Adam, RMSProp, etc.) all stayed above 0.42, at a fixed 2x10^4-step budget.
Sweeping a dial parameter from shared-scalar (equivariant) to fully coordinate-wise preconditioning showed recovery error and effective rank worsening monotonically as coordinate-wise anisotropy increased.
Varying the spectral-tail fraction tau of the target matrix showed Muon achieves exact recovery at tau=0 but is overtaken by gradient descent near tau*≈0.2 (about 4% tail energy).
In two gauge-equivalent transformer copies, Adam separated them at the very first training step and ended with per-head internal values (WQ^T WK) differing by 56%, while gradient descent and shared-scalar Adam stayed at floating-point precision throughout.
On two real hyperspectral image datasets (Indian Pines, Pavia University) compared at matched training loss, gradient descent cut held-out error by 43-44% relative to Adam at the lowest sampling density, while also reaching a lower effective rank.
Where it can be used
When training models that contain a factored structure (e.g., low-dimensional embeddings, or attention's query-key product), it may be worth considering that the optimizer choice can affect not just training speed but the complexity of the solution reached.
For underdetermined recovery or completion tasks (recovering matrices or images from partial observations), gradient-descent-family optimizers may find simpler solutions without needing explicit regularization.
When debugging or reproducing transformer attention behavior, it is worth noting that Adam can amplify tiny coordinate-basis differences into large divergences in per-head internal representations from the very first update.
Limits and open work
Experiments are limited to a synthetic 40x40 matrix recovery task, a small transformer (2 layers, 4 heads), and two hyperspectral image datasets, so the same magnitude of effect on large-scale real models is not established here.
The authors state the low-rank bias is mild and that it does not outperform tuned explicit regularization or Adam's per-coordinate adaptation on tasks that genuinely need those.
Some cases (untuned Shampoo, a long-annealed signum) are reported as exceptions to the strict two-class split, and consistency beyond n=256 problem sizes relies on separate appendix results.
One coordinate-wise method (signSGD) was reported to eventually reach comparable performance if given a much longer training budget than the 2x10^4 steps used in the main comparison, suggesting 'no access' should be read as 'slower access,' not permanent inability.
The absolute error values for GD and scalar-Adam are limited by the step-size (learning rate) choices made, so the authors emphasize the relative ranking between methods rather than the absolute numbers.
Why it matters
How a model's internal coordinates are represented is usually treated as an implementation detail, but this work shows it can determine which of many equally loss-fitting solutions an optimizer actually lands on. That matters for anyone choosing optimizers based only on training loss, since it can hide differences in the simplicity or generalization of the final solution.
Terms in this paper
gauge symmetry · the property that rotating both factors U and V of W=UV^T by the same orthogonal matrix leaves the model output and loss unchanged
gauge-equivariance · an optimizer's update rule respecting the gauge symmetry, so trajectories starting from rotated coordinates stay rotated versions of each other
low-rank bias · the tendency, under underdetermined training, to prefer simpler (lower-rank) solutions rather than arbitrary ones that also fit the data
effective rank · an entropy-based measure of how many singular directions of a matrix actually carry meaningful weight
spectral tail · the fraction of a target matrix's energy that lies outside its low-rank part
Original abstract (English)
Gradient descent on a factored model W = UV^top is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under (U, V) mapsto (UQ, VQ). Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants W_Q^top W_K 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.
