Sakana AI Unveils PC-ALM, a Learning Method That Trains Without Backpropagation

Backpropagation runs three phases in strict order: a forward pass, a backward pass, and a weight update. A neuron in an early layer has to hold its activation and wait until the error signal arrives. According to the blog, the brain has no known mechanism that could enforce this kind of timing across an entire network, which is why neuroscience has long concluded that the brain cannot implement backpropagation exactly. How the brain solves the multilayer credit assignment problem without backprop remains one of the unsolved questions in the field.

Predictive coding (PC) is the alternative that has been studied longest. It has roots in Helmholtz's theory of unconscious perception, and Rao and Ballard formalized it in 1999 as a model of the visual cortex. Each layer predicts its incoming signal on its own and passes up only the prediction error, the part it failed to model. The trouble is that the signal decays in deep, narrow networks. According to the blog, supervision entering at the output has to work its way through a chain of local compromises at every layer, and it weakens long before it reaches the early layers. The prior success the blog cites is a 128-layer residual MLP with 512 neurons per layer trained on MNIST and Fashion-MNIST.

PC-ALM adds one dual neuron, mathematically a Lagrange multiplier, to every layer. The multiplier accumulates that layer's prediction error at every step. The current error acts as the proportional term and the accumulated error as the integral term, so each layer behaves like a PI feedback controller. It works on the same principle as a car's cruise control, which presses the accelerator based on both the current speed gap and the gap that has built up over time. The starting point is a 1988 paper by Yann LeCun, which observed that the Lagrange multipliers of a constrained network equal backprop's gradients at equilibrium. Sakana AI combined that construction with PC's energy function on an augmented Lagrangian and ran layer-local dynamics on top of it. The paper proves that in bias-free linear networks the dual neurons converge exactly to backprop's signals, and confirms the nonlinear case experimentally.

The experimental numbers are recorded in the GitHub repository's README as reproduction references. According to that table, which METAL checked, training a width-32, depth-32 ReLU residual MLP on Fashion-MNIST for one epoch gave a test accuracy of 78.66% for backprop, 68.13% for standard PC, and 77.75% for PC-ALM. The cosine value measuring how closely the weight gradient points in the same direction as backprop's was 1.000 for backprop, 0.604 for PC, and 0.909 for PC-ALM. The full Fashion-MNIST grid is 675 runs. The paper compared width and depth across 8, 16, 32, 64, and 128 each, crossed with identity, tanh, and ReLU activations, and reported that with an inference budget of twice the depth, PC-ALM matched backprop across the whole range. Training ran one epoch at batch size 64 on two datasets, MNIST and Fashion-MNIST. The 1,000-layer experiment is a new result that appears on the blog. It ran five epochs, with 2,000 inference steps, twice the number of layers. The company also wrote that PC-ALM beat PC when training ResNet-18 on CIFAR-10 and Tiny ImageNet.

What the authors described as surprising is the shape in which the signal spreads. In PC, supervision diffuses like heat and decays with depth, while in PC-ALM it advances quickly from the output toward the input like a wavefront and spreads evenly across the whole network. The authors called this ballistic propagation, in contrast to diffusive propagation. Individual neurons show damped oscillations during inference. When the dual step size α is 0, the method reduces exactly to standard PC; increasing α introduces complex eigenvalues and oscillation, and increasing it too far destabilizes the network, a condition the paper states as an equation.

Seen through a sociologist's eyes, this research is also a bridge between two academic communities. The authors said they started from the observation that the NeuroAI community on the neuroscience side and the distributed optimization community on the optimization side both care about locality yet have had little cross-talk. A second observation, that PC's energy looks suspiciously like the augmented term of the augmented Lagrangian commonly used in distributed optimization, joined the two fields. The cost is written down in numbers. According to the paper, PC-ALM uses twice as much activation memory as PC and has to run an inference budget proportional to depth on every mini-batch. In exchange, compared with the PC approach of widening the network to approach backprop, the authors calculated that reaching a cosine of 0.9 to the backprop gradient takes an order of magnitude less computation. The company named neuromorphic hardware as where this work could be useful. On chips where simulating dynamics is cheaper than on a GPU, a learning method in which each layer communicates only with its neighbors could lead to energy-efficient deep learning. The experiments stayed within MNIST, Fashion-MNIST, and residual MLPs and were not applied to transformers or language models, and the authors wrote in the paper's appendix that the weight transport problem, in which the energy gradient still uses the transpose of the same layer's weights, is not solved by this method. METAL has reported that Sakana AI introduced the Royal Society's theme issue on world models, and research asking how the brain learns continues at the company.

The authors, Jeffrey Seely and Julian Gould of Sakana AI, wrote on the blog: "To our knowledge, this is the first layer-local method shown to successfully train networks up to 1000 layers." In its X post the company said: "Ultimately, our motivation here is to understand how distributed physical systems, such as the brain, can compute credit signals using only local coupling and local dynamics." The post passed 60,000 views within hours of going up, and the attached 45-second silent video illustrates the difference between backprop, PC, and PC-ALM. The authors listed temporal credit assignment, self-supervised losses, and larger networks with harder tasks as future work.

Deep learning today rests on backpropagation alone. That a network in which each layer talks only to its neighbors came within two percentage points of backprop at 1,000 layers is the first evidence that brain-inspired learning can move beyond toy sizes. The next number to watch is how little energy this method actually uses on a neuromorphic chip.