Gradient

Researchers at QLabs have published Dust, a method for pretraining transformer models without using backpropagation. The approach, described in a new research paper, would replace the gradient-based training that underpins virtually all modern deep learning. Interest is concentrated among machine learning practitioners debating whether such alternatives could realistically match conventional training at scale.