Qlabs

Researchers at Qlabs have presented Dust, a method for pretraining transformer models without using backpropagation. The work, described in a research note from qlabs.sh, suggests an alternative to the gradient-based training that underpins virtually all modern deep learning. The approach is drawing attention from machine learning practitioners debating whether backprop-free training could reduce the cost or energy demands of building large language models.

A research project called Dust claims a method for pretraining transformer models without backpropagation, the algorithm at the core of modern deep learning. If the results hold up, the approach could challenge assumptions about how large models must be trained, but independent verification and details of its performance are not yet established.