singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangwei (JIRA)" <>
Subject [jira] [Created] (SINGA-8) Implement distributed Hogwild
Date Sat, 13 Jun 2015 06:47:00 GMT
wangwei created SINGA-8:

             Summary: Implement distributed Hogwild
                 Key: SINGA-8
             Project: Singa
          Issue Type: New Feature
            Reporter: wangwei
            Assignee: wangwei

Generally, both the Downpour framework of Google Brain [1] and the Caffe's distributed Hogwild
implementation are extensions of the shared memory Hogwild training. In this ticket, we refer
to the second one.

In specific, each server group masters a subset of parameters (i.e., Param objects) when synchronizing
with other server groups. It aggregates all updates for its subset and sends back (e.g., broadcast)
the updated parameters back to all other server groups. The synchronization is conducted asynchronously.
The frequency can be fixed in the first implementations. Finally, it should be tuned automatically
to fully utilize the network bandwidth.

[1]J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W.
Senior, P. A. Tucker, K. Yang, and A. Y. Ng. Large scale
distributed deep networks. In NIPS, pages 1232{1240, 2012.

This message was sent by Atlassian JIRA

View raw message