singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] nudles opened a new issue #674: Autograd Layer constructor
Date Sat, 11 Apr 2020 15:04:16 GMT
nudles opened a new issue #674: Autograd Layer constructor
URL: https://github.com/apache/singa/issues/674
 
 
   The Layer class in Autograd is to maintain the model parameters.
   It passes the parameters into the operation and thus operations are stateless.
   
   Typically the parameter size depends on the input and layer configuration.
   Currently, we require the users to provide the input size in the layer constructor.
   Then we can create the parameter tensor and initialize it in the constructor, e.g., [Linear
layer](https://github.com/apache/singa/blob/master/python/singa/autograd.py#L1387). One potential
problem is that the initialization operation may not be buffered. @XJDKC  Is it an issue?
   For some layers like RNN implemented using cudnn, although we can get the input size, the
parameter size is unknown until the cudnn handle is created, which is done until the data
is forwarded through the layer.
   
   Another way is to delay the parameter tensor creation until the layer is called for forward
propagation. At that time, we have the input tensor (and its device). Then in the layer constructor,
we do not need the user to provide the input size. The drawback is that after the layer is
created, the get_params() function would still fail to get the parameter tensors as they are
not created yet. @dcslin  To switch to this approach, we need to change the constructors of
existing layer classes and examples. We also need to provide an initializer function/class
into the constructor for initializing the parameter tensors after they are created.
   
   Please add your comments.
   
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message