singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] chrishkchris edited a comment on issue #591: Dev branch cpu training problem (with conv and pool)
Date Thu, 20 Feb 2020 06:58:24 GMT
chrishkchris edited a comment on issue #591: Dev branch cpu training problem (with conv and
pool)
URL: https://github.com/apache/singa/issues/591#issuecomment-588630599
 
 
   > Hi @chrishkchris , I am still working on this issue, but I am encountering something
very weird.
   > I borrow the code from dnnl example and testing result shows similar performance compared
to mkldnn(conv forward in 5000microsec).
   > However, after some modification to the example code(just introducing Singa Tensor
dcslin/singa@06ec6ce), I still can get reasonable performance(conv forward in 5000 microsec).
But the weird part is, the result is not stable, out of 10 times testing, segmentation fault
appear ~6 times.
   > 
   > I run your example code but did not find any problem, it display mytestok. The last
segmentation fault of "double free" is due to freeing of variables.
   > [ RUN      ] MYTEST.Forward
   > [total]Time difference = 58228[mu s]
   > [avg]Time difference = 582[mu s]
   > mytestok
   > double free or corruption (!prev)
   > Aborted (core dumped)
   > 
   > 
   > do you how to fix this?
   
   OK now, I tried by turning the tensor to pointer. 
   
   ```
     Tensor *in = new Tensor(Shape{batch, in_chan, image_h, image_h});
     Tensor *out = new Tensor(Shape{batch, out_chan, out_size, out_size});
     Tensor *weights = new Tensor(Shape{out_chan, in_chan, ker, ker});
     Tensor *bias = new Tensor(Shape{out_chan});
     in->SetValue(1.0f);
     weights->SetValue(1.0f);
     bias->SetValue(1.0f);
   ```
   
   Now it display:
   ```
   [----------] 1 test from MYTEST
   [ RUN      ] MYTEST.Forward
   [total]Time difference = 51943[mu s]
   [avg]Time difference = 519[mu s]
   mytestok
   [       OK ] MYTEST.Forward (53 ms)
   [----------] 1 test from MYTEST (53 ms total)
   
   ```
   so I suspect that when it destruct dnnl memory object, it will free the memory from in->block()->mutable_data
… 
   For the same reason, I suspect that in the constructor of dnnl memory object, it will reallocate
memory based on the pointer given.
   
   The solution is to avoid passing the block memory pointer to the dnnl memory constructor,
instead we copy the block memory to the dnnl memory after dnnl memory object is constructed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message