Accelerating Gossip SGD with Periodic Global Averaging

Communication overhead hinders the scalability of large-scale distributed training. Gossip SGD, where each node averages only with its neighbors, is more communication-efficient than the prevalent parallel SGD. However, its convergence rate is reversely proportional to quantity $1-β$ which measures...

Full beskrivning

Bibliografiska uppgifter

Huvudupphov:	Chen, Yiming, Yuan, Kun, Zhang, Yingya, Pan, Pan, Xu, Yinghui, Yin, Wotao
Materialtyp:	Artikel
Utgiven:	2021
Ämnen:	Machine Learning Distributed, Parallel, and Cluster Computing
Länkar:	https://arxiv.org/abs/2105.09080 http://arxiv.org/abs/2105.09080
Taggar:	Lägg till en tagg Inga taggar, Lägg till första taggen!