singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] dcslin commented on issue #591: Dev branch cpu training problem (with conv and pool)
Date Thu, 20 Feb 2020 07:02:28 GMT
dcslin commented on issue #591: Dev branch cpu training problem (with conv and pool)
URL: https://github.com/apache/singa/issues/591#issuecomment-588648313
 
 
   @chrishkchris , in fact, even disabling reorder part, the performance is still the same
   
   > > Hi @chrishkchris , I am still working on this issue, but I am encountering something
very weird.
   > > I borrow the code from dnnl example and testing result shows similar performance
compared to mkldnn(conv forward in 5000microsec).
   > > However, after some modification to the example code(just introducing Singa Tensor
[dcslin/singa@06ec6ce](https://github.com/dcslin/singa/commit/06ec6ce)), I still can get reasonable
performance(conv forward in 5000 microsec). But the weird part is, the result is not stable,
out of 10 times testing, segmentation fault appear ~6 times.
   > > I run your example code but did not find any problem, it display mytestok. The
last segmentation fault of "double free" is due to freeing of variables.
   > > [ RUN      ] MYTEST.Forward
   > > [total]Time difference = 58228[mu s]
   > > [avg]Time difference = 582[mu s]
   > > mytestok
   > > double free or corruption (!prev)
   > > Aborted (core dumped)
   > > do you how to fix this?
   > 
   > OK now, I tried by turning the tensor to pointer.
   > 
   > ```
   >   Tensor *in = new Tensor(Shape{batch, in_chan, image_h, image_h});
   >   Tensor *out = new Tensor(Shape{batch, out_chan, out_size, out_size});
   >   Tensor *weights = new Tensor(Shape{out_chan, in_chan, ker, ker});
   >   Tensor *bias = new Tensor(Shape{out_chan});
   >   in->SetValue(1.0f);
   >   weights->SetValue(1.0f);
   >   bias->SetValue(1.0f);
   > ```
   > 
   > Now it display:
   > 
   > ```
   > [----------] 1 test from MYTEST
   > [ RUN      ] MYTEST.Forward
   > [total]Time difference = 51943[mu s]
   > [avg]Time difference = 519[mu s]
   > mytestok
   > [       OK ] MYTEST.Forward (53 ms)
   > [----------] 1 test from MYTEST (53 ms total)
   > ```
   > 
   > so I suspect that when it destruct dnnl memory object, it will free the memory from
in->block()->mutable_data …
   > For the same reason, I suspect that in the constructor of dnnl memory object, it will
reallocate memory based on the pointer given.
   > 
   > The solution is to avoid passing the block memory pointer to the dnnl memory constructor,
instead we copy the block memory to the dnnl memory after dnnl memory object is constructed.
   
   will it cost double memory usage?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message