singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SINGA-315) Reduce memory footprint by Python generator for parameter gradient
Date Wed, 24 May 2017 12:13:04 GMT

    [ https://issues.apache.org/jira/browse/SINGA-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022788#comment-16022788
] 

ASF subversion and git services commented on SINGA-315:
-------------------------------------------------------

Commit ea078dca9cfafdda9d5a15ae2f2823897d217292 in incubator-singa's branch refs/heads/master
from wangwei
[ https://git-wip-us.apache.org/repos/asf?p=incubator-singa.git;h=ea078dc ]

SINGA-315 Reduce memory footprint by Python generator for parameter gradient

Update the API of net::backward() function.
1. add arguments dy and output. dy for the input gradient tensor(s), e.g. from the loss functions.
   output is a list of layer names, whose output gradient tensor(s) would be returned in addition
to the param gradient tensor(s).
2. returnes a generator iterator that generates (param_names, param_values, param_grads, out_grads)
after processing each layer.
The callee function can update the parameters and release the gradient tensors in layerwise.


> Reduce memory footprint by Python generator for parameter gradient
> ------------------------------------------------------------------
>
>                 Key: SINGA-315
>                 URL: https://issues.apache.org/jira/browse/SINGA-315
>             Project: Singa
>          Issue Type: New Feature
>            Reporter: wangwei
>
> The parameter gradient tensors are stored in memory until BP is finished.
> This is not necessary as we can update the parameter once its gradient is ready. Then
we free the gradient tensor. 
> In this way, we reduce the memory footprint by avoiding store all gradient tensors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message