Figure 1: Equivariance structures the set of available optimizers under a fixed computational budget. Ground-truth recovery for nine optimizer rules (eight standard rules plus a shared-scalar Adam as a baseline) on an underdetermined matrix sensing task without weight decay. All are run until the training residuals vanish (interpolation), which rules out any performance difference due to residual training error. The four gauge-equivariant methods (blue) maintain the low-rank bias while the five coordinate-wise approaches (vermilion) do not: on this task and at this budget, recovery error is at most 0.286 for every equivariant method and above 0.42 for every coordinate-wise one, leaving the space between the two unoccupied. That qualifier matters, as one of the coordinate-wise methods eventually reaches similar performance beyond the 2×104-step budget used here (Appendix D.10). Muon’s label reads 6.8×10−6, the unrounded value of the 0.0000 entry in Table 2.
Table 1: Closest prior and concurrent work. The recurring distinction is whether an update respects a relevant orthogonal symmetry; we ask what it predicts for interpolant selection in factored models at matched training loss.
work
setting and result
relation to this paper
VectorAdam; Xie et al. 2025
Examine ambient rotations of vector parameters under general loss functions. Show Adam is sensitive to such rotations, whereas shared scalar updates restore equivariance.
We focus instead on an internal factorization gauge and connect the same principle to solution selection among interpolants, rather than optimization speed.
LoRA-RITE; DePavia et al. 2025
Study transformations of LoRA factors and rotations in data or feature space. Demonstrate improved optimization through invariance and altered implicit bias in Adam under rotations.
We isolate the intrinsic gauge of the factorization itself and evaluate recovery performance at matched training loss across a range of optimizers.
Silverstein et al. 2026; Zhang 2026
Analyze per-head query–key (QK) rotations and Schur multiplicity bases. Highlight that optimizer design can intentionally break or preserve symmetry.
We maintain the gauge structure intact and use it to analyze solutions chosen by standard, deployed optimizers.
Lau & Su 2026; Shirodkar 2026
Investigate architectural symmetry groups and construct optimizers that exactly respect equivariance.
These works support symmetry-aware design; we instead assess the practical consequences of preserving or violating such symmetry.
Kang et al. 2026; Dragutinović et al. 2026
Study Muon’s spectral dynamics and its impact on simplicity bias.
We extend this by mapping conditions under which uniform spectral growth helps or harms performance as the target develops a spectral tail.
Figure 2: Adam is basis-dependent in attention, shown here by the relative logit distance between twins in the gauge and noise experiments. In the former the two copies are initialized as the same function, while in the latter they differ by a small (10−7) perturbation in the same basis. Adam’s gauge twins (solid vermilion line) split structurally in one step, to a value four orders of magnitude greater than its own noise twin (dotted line), and then saturate. The same is not observed for SGD (heavy-ball momentum 0.9) or scalar-Adam, whose gauge twins remain at float noise. For Muon the gauge and noise twins trace the same curve, indicating that its separation is numerical chaos stemming from msign, not basis dependence.
Table 2: The optimizer-zoo map. Matrix sensing, 40×40, rank 3 ground truth, m=2×dof, no weight decay, three paired random seeds, all nine optimizer runs continued until interpolation (final training loss reported). Recovery is measured as ‖W−X∗‖F/‖X∗‖F (lower is better); erank is effective rank; bal is ‖U⊤U−V⊤V‖F; the notes column names the theoretical framework each row falls under—Gunasekar et al. 2017 for gradient descent (GD) and Wilson et al. 2017 for Adam. (cosine) marks the four methods that need cosine annealing to interpolate; the first row is a convex baseline, not an optimizer, recovering the minimum-nuclear-norm solution of the same measurements (both are unpacked in the text above). All values are means over the three seeds, with no dispersion quoted.
method
recovery ↓
erank
bal
train loss
notes
min-nuclear-norm
0.0335
3.30
—
—
convex reference, not an optimizer
equivariant
Muon (cosine)
0.0000
2.95
1.65
5.7×10−8
near-exact (6.8×10−6 unrounded)
GD
0.1312
4.51
0.06
6.5×10−8
Gunasekar anchor
scalar-Adam (p=0)
0.2010
5.43
0.14
3.5×10−8
equivariant; flow proxy
Shampoo
0.2856
6.95
1.47
4.5×10−8
coord.-wise
Lion (cosine)
0.4248
10.58
7.35
8.0×10−8
signum (cosine)
0.4454
7.83
1.79
6.5×10−8
low-rank-but-wrong
RMSProp (cosine)
0.5266
12.80
4.74
6.0×10−8
needs decay; Appendix D.7
Adafactor
0.5430
10.72
3.40
5.6×10−8
factored diag. still breaks
Adam
0.5734
14.37
5.37
1.2×10−11
Wilson anchor
Figure 4: The spectral-tail phase diagram. Recovery vs. the target’s tail-amplitude parameter τ (the tail carries τ2 of the target’s energy). Adam has the worst recovery error at every τ plotted. Within the equivariant class, Muon’s aggressive equal-rate schedule is exact at τ=0 and crosses over to GD near τ∗≈0.2. At the largest τ, all methods are near the measurement-determined floor set by the unidentifiable tail, and GD and Shampoo are within seed noise, so the ordering there is uninformative. Bands show ±1 s.d. over 10 seeds. The boundary is unchanged when every method is given the identical cosine schedule (Appendix D).
Table 3: Adam is basis-dependent in attention, and the noise twin separates structure from chaos. The values are relative logit distances between twins on validation inputs for the same task; multi-seed statistics are shown in Appendix D.5. At the harness’s default CPU precision, the baseline rounding threshold is the step-0 column (≈2×10−7, i.e. one function evaluated across two bases); the equivariant methods reach this floor at step 1, and their later nonzero entries result from numerical noise, not from gauge-symmetry violations—Muon’s split in particular, as per Proposition 4.2. These discrepancies are reduced, if not eliminated, in GPU float64, where the step-1 values fall within 2.8×10−16–1.1×10−15 over the three methods and four configurations; Table 9 carries the worst case in each.
optimizer
twin type
step 0
step 1
step 100
step 1500 (final)
Adam
gauge
1.8×10−7
3.6×𝟏𝟎−𝟑
6.5×10−1
7.7×10−1
Adam
A=I
0
0
0
0
Adam
noise (10−7)
2.6×10−7
2.9×10−7
1.7×10−5
1.6×10−5
SGD
gauge
1.8×10−7
2.9×10−7
3.7×10−5
2.0×10−5
scalar-Adam
gauge
1.8×10−7
1.6×10−7
4.1×10−6
4.3×10−6
Muon
gauge
1.8×10−7
2.2×10−7
2.3×10−2
8.5×10−1
Muon
noise (10−7)
2.6×10−7
2.9×10−7
4.0×10−2
8.7×10−1
Figure 5: Performance as a function of training loss on real data (Indian Pines, m/dof≈1.15, 4 random seeds, GPU float64; learning rates chosen via the train-only rule in Appendix D.6). This is the corresponding row of Table 5, read along its whole path rather than at its endpoint alone. Gradient descent (GD) sits below Adam at every matched training loss beyond the shared starting point, and the two are compared only at equal training loss, so no fitted run is ever set against an unfitted one. Left panel: held-out error; bands are ±1 s.d. over the 4 seeds, hidden inside the line width wherever the seeds agree—the whole path for GD—and opening up only where they do not, as Adam’s do once overfitting begins. Right panel: effective rank. GD settles between 5.6 and 13.2; Adam climbs to 28, against the scene’s intrinsic rank of 24; Muon tracks the rank cap until convergence, then drops sharply.
Table 4: Spectral-tail phase diagram (recovery; 10 seeds, float64). Muon is exact at τ=0, degrades fastest as tail energy grows, and cedes to GD in a crossover region near τ∗≈0.2 (≈4% tail energy). Bold marks each row’s best recovery, with both members of a within-noise tie bolded; the seed dispersions behind those ties are the ±1 s.d. bands of Figure 4, omitted here for space. Decay-symmetrized rows in Appendix D.
τ
GD
Adam
Muon
Shampoo
regime
0.00
0.112
0.542
0.000
0.334
Muon exact
0.05
0.150
0.543
0.095
0.343
Muon
0.10
0.214
0.555
0.191
0.375
Muon
0.20
0.351
0.597
0.354
0.449
boundary (≈4% tail); Muon cedes
0.35
0.549
0.685
0.580
0.576
GD
0.50
0.727
0.776
0.752
0.723
tail regime; GD/Shampoo within noise
Table 5: Matched-loss recovery on two hyperspectral datasets (held-out RMSE ×10−2, mean ± standard deviation over 4 random seeds, evaluated at matched training loss ≤10−5, GPU float64; learning rates as in Appendix D.6). The reduction column reports the percent improvement of GD over Adam, as (Adam−GD)/Adam using mean performance over seeds; gradient descent (GD) outperforms Adam in every individual seed of every entry. In parentheses, Muon’s effective rank at the deepest point in the fit (after the collapse typical of the late stages), given a model capacity of 48.
dataset
m/dof24
GD
Adam
RMSE reduction
Muon (rank)
Indian Pines
1.15
1.481±0.013
2.600±0.047
+43.0%
3.397±0.082 (36)
1.9
1.244±0.012
1.718±0.035
+27.6%
3.210±0.122 (35)
Pavia Univ.
1.15
0.819±0.026
1.458±0.123
+43.8%
1.260±0.121 (10)
1.9
0.615±0.010
0.798±0.023
+22.9%
0.770±0.017 0(8)
Table 6: Recovery under three schedules (3 seeds, best-recovery lr per cell) for five reference methods: the equivariant anchors GD, scalar-Adam, and Shampoo, the coordinate-wise anchor Adam, and the constant-norm case Muon. The split does not move; under a single uniform cosine schedule the complete nine-method classification is again 9/9 (selection-rule test, Appendix D.3).
method
constant
cosine
cosine, full horizon
GD
0.1312
0.1312
0.1312
Adam
0.5734
0.5734
0.5764
scalar-Adam
0.2010
0.2010
0.2010
Muon
(0.0084†)
0.0000
0.0000
Shampoo
0.2856
0.2856
0.2856
Table 7: Recovery versus learning rate for the five reference methods, averaged over three random seeds. For each method the results occupy two rows: the top row gives the learning-rate grid used (of varying length, since Adam has five rates while the rest have four), and the second row the corresponding recovery, so a method’s row simply ends where its grid does. “→fl” signifies the lowest learning rate at which interpolation still succeeds, the flow-limit-aligned column; n/i = does not interpolate; div = diverges.
method
learning-rate grid
GD
lr
0.003→fl
0.01
0.03
0.1
recovery
0.113
0.131
0.437
div
Adam
lr
0.0003→fl
0.001
0.003
0.01
0.03
recovery
0.560
0.576
0.570
0.573
0.581
scalar-Adam
lr
0.0003→fl
0.001
0.003
0.01
recovery
0.165
0.201
0.256
0.357
Muon (cosine)
lr
0.003
0.01
0.03
0.1
recovery
0.915
0.748
0.000
0.000
Shampoo
lr
0.01
0.03
0.1
0.3
recovery
(0.881 n/i)
0.286
div
div
Table 8: Recovery vs. initialization scale (3 seeds). The split holds at 10−3 (the paper’s setting) and 3×10−3; at 10−2 the small-init bias fades for every method except Muon, which remains exact, marking its bias as schedule- rather than init-driven.
init
GD
Adam
scalar-Adam
Muon
Shampoo
10−3
0.131
0.573
0.201
0.000
0.286
3×10−3
0.203
0.575
0.260
0.000
0.352
10−2
0.318
0.584
0.358
0.000
0.575
Table 9: Twin drift measured at scale and on real text, in GPU float64 precision. For each configuration, Adam runs 6 init×draw gauge pairs while each equivariant method runs 2; the A=I determinism twin is evaluated for Adam at every configuration and is exactly 0 throughout. “eq. step 1” is the maximum (worst) gauge drift across the three equivariant methods (SGD, scalar-Adam, Muon) at step 1, all at machine precision; “ratio” is the onset ratio defined above. The character-level language model twins use the same deterministic minibatch stream and reach comparable validation losses (1.579 vs. 1.585).
config
Adam gauge, step 1
eq., step 1
Adam, final
ratio (step 1)
mod-47, 2L, d=64
4.0–5.7×10−3
5.4×10−16
0.69–0.78
93×
mod-97, 4L, d=128
8.2–11×10−3
7.3×10−16
0.66–0.75
158×
mod-97, 6L, d=256
6.5–7.2×10−3
1.1×10−15
0.61–0.64
94×
text (char-LM), 6L, d=256
2.2–2.4×10−3
8.0×10−16
0.36–0.37
39×
Table 10: FlowAdam-p=0 recovery along a trajectory aligned by training loss (extended 1.2×105-step budget, 3 seeds). The mean at the 10−7 threshold (0.1691) matches the original 3×104-step result exactly and improves upon the dial-only baseline (0.2010) by +15.9%. Its effective rank (4.8–5.4) is the closest of any Adam variant in this study to gradient descent’s (4.51).
train loss ≤
10−4
10−5
3×10−6
10−6
10−7
erank
seed 42
0.1404
0.1401
0.1401
0.1401
0.1401
4.77
seed 123
0.1558
0.1556
0.1555
0.1555
0.1555
4.84
seed 456
0.2120
0.2117
0.2117
0.2117
0.2117
5.41
mean
0.1694
0.1691
0.1691
0.1691
0.1691
5.01
Table 11: Optimization algorithms on the Zoo recovery task, across problem sizes (10-seed averages; float64, H100 GPU, 4×104 steps). The ranking is consistent across scales: gradient descent (GD), scalar-Adam (p=0), and Muon significantly outperform coordinate-wise Adam, RMSProp, Lion, and Adafactor, with Adam-type methods at 1.7–4.3× GD’s error at every size and Muon almost exact (error <10−5 for all seeds) up to n=256. The errors of GD and scalar-Adam are limited by the step-size choice (Remark A.5), so the relative ordering, not the absolute value, is the conclusion.
Gradient descent on a factored model W = UV^top is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under (U, V) mapsto (UQ, VQ). Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants W_Q^top W_K 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.
