singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangwei (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SINGA-71) Fix a buf from repeated averaging gradients from workers
Date Sat, 26 Sep 2015 15:14:04 GMT

     [ https://issues.apache.org/jira/browse/SINGA-71?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

wangwei resolved SINGA-71.
--------------------------
    Resolution: Fixed

> Fix a buf from repeated averaging gradients from workers
> --------------------------------------------------------
>
>                 Key: SINGA-71
>                 URL: https://issues.apache.org/jira/browse/SINGA-71
>             Project: Singa
>          Issue Type: Bug
>            Reporter: wangwei
>            Assignee: wangwei
>
> For workers from the same group, if they share parameters, gradients of parameters should
be averaged. SINGA averages the gradients by
> * aggregates gradients from all workers
> * calls Updater::Update() passing the scale_grad = 1.0 / worker_group_size. 
> * The Update function multiplies the gradient with scale_grad.
> The bug was from the Trainer::HandleUpdate() which averaged the gradients from local
workers. The bug has been fixed by removing the averaging operation. Gradients from local
workers are (only) summed before sending to servers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message