Does compressing activations help model parallel training?

Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approa...

Disgrifiad llawn

Manylion Llyfryddiaeth

Prif Awduron:	Bian, Song, Li, Dacheng, Wang, Hongyi, Xing, Eric P., Venkataraman, Shivaram
Fformat:	Erthygl
Cyhoeddwyd:	2023
Pynciau:	Machine Learning
Mynediad Ar-lein:	https://arxiv.org/abs/2301.02654 http://arxiv.org/abs/2301.02654
Tagiau:	Ychwanegu Tag Dim Tagiau, Byddwch y cyntaf i dagio'r cofnod hwn!