紀要論文

Decentralized Deep Learning with Gradient Compression

eng

24321192

https://doi.org/10.15002/00021917

法政大学大学院情報科学研究科

法政大学大学院紀要. 情報科学研究科編

14

1

6

2019-03-31

Version of Record

Training a deep neural network (DNN) with a single machine consumes much time. To accelerate the training, a popular method is distributed deep learning (DDL), that is to splits and distributes DNN training tasks on multiple machines. One of the common designs of the DDL is a centralized architecture, that contains one parameter server and several workers. In this architecture, several workers need communicate with the parameter server to get the latest parameter at every iteration. The network resource of the parameter server will be exhausted frequently, which further lead to the limitation of the total training performance. In this paper, we introduce a decentralized approach for distributed deep learning. Our proposed decentralized DDL removes the parameter server and all worker nodes are designed to communicate with each other directly to exchange the intermediate results of their own model. For good generalization, we built a customized optimizer class to compress the every iteration’s gradient that only update the gradient larger than threshold value. Compared with centralized distributed approach, our proposed model gains better accuracy during a whole test. When we limit the parameter server’s bandwidth to 300Mbps in order to simulate the network bottleneck problem, the performance of our proposed decentralized DDL improved several times faster than the existing centralized approach on the worst case execution time.

Article

AA12746425