singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SINGA-226) Add parallel training on a single machine for singa v1.0
Date Fri, 22 Jul 2016 13:25:20 GMT

    [ https://issues.apache.org/jira/browse/SINGA-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389482#comment-15389482
] 

ASF subversion and git services commented on SINGA-226:
-------------------------------------------------------

Commit 4e7f3c13b1de9bbe58d25dccec5d67ac8d98a3fe in incubator-singa's branch refs/heads/dev
from WANG Ji
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=4e7f3c1 ]

SINGA-226 Add parallel training on a single machine for singa v1.0

Update local_updater.cc to an alternative version which
fully utilizes parallelly data copying.


> Add parallel training on a single machine for singa v1.0
> --------------------------------------------------------
>
>                 Key: SINGA-226
>                 URL: https://issues.apache.org/jira/browse/SINGA-226
>             Project: Singa
>          Issue Type: New Feature
>            Reporter: Wang Ji
>            Assignee: Wang Ji
>
> In this ticket, we implement parallel training using multiple devices on a single machine.

> To support parallel training, a Updater class need to be implemented to aggregate partial
gradient from parallel workers and using Optimizer to update the Parameters. Updater can be
designed for different kinds of topological structure, i.e., *local-cpu*, *local-dev*, *local-allreduce*.

> *local-cpu:* Do aggregate and update parameter using CPU. In this mode, host CPU need
to copy gradient and parameter tensor from GPU workers, do update, and copy back.
> *local-gpu:* Do aggregate and update parameter using a chosen GPU. In this mode, the
updater GPU need to copy gradient and parameter tensor from other GPU workers, do update,
and copy back.
> *local-allreduce:* In this mode, each parameter will be sliced among all GPU workers.
In each iteration, gradients are aggregated and updated like a MPI Allreduce style.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message