singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-singa] chrishkchris commented on issue #468: Distributted module
Date Wed, 14 Aug 2019 13:32:43 GMT
chrishkchris commented on issue #468: Distributted module
URL: https://github.com/apache/incubator-singa/pull/468#issuecomment-521247206
 
 
   > Just now I did a test: When I replace the merged tensor.cc file by the latest master
branch tensor.cc, the loss can be reduced again. So I just changed the tensor.cc of dist_new
to the latest master branch one. This act would omit the commit f54a526 done on tensor.cc
   > 
   > I guess this maybe because:
   > The commit f54a526 in dist_new has redesigned the softmax function by calling cudnn,
but this is not compatiable with the latest master branch.
   
   OK, I check again in the AWS image, recently the code we used for distributed module has
commented out the code done by the commit f54a526 on softmax function, so softmax is the same
as master branch now. This is the code of softmax we used in the AWS image of distributed
module:
   ```
   void SoftMax(const Tensor &in, Tensor *out) {
     CHECK_LE(in.nDim(), 2u);
     out->CopyData(in);
     size_t nrow = 1, ncol = in.Size(), size = ncol;
     if (in.nDim() == 2u) {
       nrow = in.shape(0);
       ncol = size / nrow;
       out->Reshape(Shape{nrow, ncol});
     }
     Tensor tmp = RowMax(*out);
     SubColumn(tmp, out);
     Exp(*out, out);
   
     SumColumns(*out, &tmp);
     DivColumn(tmp, out);
     out->Reshape(in.shape());
   }
   
   // void SoftMax(const Tensor &in, Tensor *out) {
   //   CHECK_LE(in.nDim(), 2u);
   //   TYPE_LANG_SWITCH(in.data_type(), DType, in.device()->lang(), Lang, {
   //     out->device()->Exec([in, out](Context * ctx) {
   //       SoftMax<DType, Lang>(in, out, ctx);
   //     }, {in.block(), out->block()}, {out->block()});
   //   });
   // }
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message