Q Labs has published Dust, a method that trains transformer language models by testing random changes to their internal activations instead of using backpropagation. The researchers report that it can approach, and in some settings beat, backpropagation’s prediction loss. Those results require substantially more compute, and the authors say practical use needs orders of magnitude better compute efficiency.
Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna describe the work as a foundation for search-based training. The distinction matters for anyone reading the headline as a route to cheaper model training.
How Dust assigns credit
Backpropagation calculates how each weight contributed to a model’s prediction error. Dust estimates useful weight updates by adding random noise to activations, the values that a layer produces during a forward pass. It rewards perturbations that lower the loss, then averages the reward-weighted noise to estimate an update.
The method gives each token its own perturbation. That lets one forward pass test many perturbations in parallel, which the authors call a virtual population. A weight-space search method instead evaluates separate perturbed versions of the model’s weights.
Attention needs extra handling because a change at one token can affect later tokens. Dust uses estimated errors at the attention output to score changes inside attention. To reduce interference between perturbations, it also separates layer types into different passes and reuses cached clean activations. Training therefore involves more than a single forward pass per update.
What the experiments show
The main experiments train GPT-style transformers on FineWeb, with token budgets between 100,000 and 20 million and populations between 64 and roughly 16,000 draws per update. The researchers tune each method separately and report test loss at the checkpoint with the best validation loss, using three seeds per setting.
Dust beats the tuned backpropagation baseline at sufficiently large populations for the 100,000-token and 1-million-token budgets. At 10 million and 20 million tokens, the gap shrinks as the population grows. At 20 million tokens, a fitted curve predicts an eventual loss below backpropagation’s, but the authors call that limit loosely constrained. It is an extrapolation beyond the measured results.
A separate size comparison trains models between 2 million and 243 million parameters on a fixed 10 million tokens. The largest model beats the smallest at most population sizes, though the 38-million-parameter model does better than the largest across the reported Dust settings. More parameters do not automatically make the method better.
The authors also report an estimated efficiency advantage of roughly 1,000 to 10,000 times over their transformer implementation of EGGROLL, a weight-space evolution strategy. That estimate relies on extrapolating how much population EGGROLL would need to match Dust.
Sources
- Q Labs: Dust: Pretraining Transformers Without Backpropagation. Research and lead diagram by Q Labs. The lead shows a still from its method animation.
