singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] nudles commented on issue #696: Refactor autograd module
Date Thu, 14 May 2020 03:31:05 GMT

nudles commented on issue #696:
URL: https://github.com/apache/singa/issues/696#issuecomment-628365342


   > 
   > 
   > > 1. Before we call Module.forward(), we can randomly fill the placeholder tensors.
   > > 2. We can make Layer.init() optional. To implement a new layer, the parameter
initialization can be done within the `__call__` method or in a `init()` method. It is up
to the contributor.
   > > 
   > > Any comments on the drawbacks?
   > > @dcslin @XJDKC
   > 
   > For some models, it cannot use the random inputs, such as BERT within ONNX, some nodes
may compute the indices of a tensor, and the next node may split the tensor by using these
indices. If we randomly generate the inputs, this case always fails.
   Good point. Then we can config the data type when creating the placeholder and initialize
the placeholder according to this data type. But how to initialize? randomly or set to 0?
there could still be some issues.
   > 
   > By the way, I prefer the idea of:
   > 
   > ```python
   > # another option is to define a compile method
   >     def compile(self, inputs, is_train, use_graph, graph_alg):
   >         self.forward(*inputs)
   > ```
   > 
   > However, I'd like to add a method to compute the shape based on the inputs of each
node instead of calling the forward function:
   > 
   > ```python
   > def compute_output_shape(self, input_shape):
   >     # print(input_shape) # [(None, 10), (None, 12)]
   >     return (None, input_shape[0][1] + input_shape[1][1] + 2)
   > ```
   > 
   > Let me think about it, I'll comment the detailed API later.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message