singa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [singa] joddiy commented on issue #807: AssertionError for the ONNX training testcases?
Date Tue, 13 Oct 2020 11:01:37 GMT

joddiy commented on issue #807:
URL: https://github.com/apache/singa/issues/807#issuecomment-707664028


   > AssertionError with the onnx testcase: https://github.com/apache/singa/blob/master/examples/onnx/training/train.py
   > 
   > ```
   > $ cd examples/onnx
   > $ python3 training/train.py --model vgg16
   > ```
   > 
   > Then I get the following error msg:
   > 
   > ```
   > File "training/train.py", line 437, in <module>
   >     args.onnx_model_path, args.data, sgd, args.graph, args.verbosity)
   >   File "training/train.py", line 295, in run
   >     model.compile([tx], is_train=True, use_graph=graph, sequential=sequential)
   >   File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/model.py",
line 177, in compile
   >     self.forward(*inputs)
   >   File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py",
line 63, in wrapper
   >     return func(self, *args, **kwargs)
   >   File "training/train.py", line 191, in forward
   >     y = self.linear(y)
   >   File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py",
line 110, in __call__
   >     return self.forward(*args, **kwargs)
   >   File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py",
line 61, in wrapper
   >     self.initialize(*args, **kwargs)
   >   File "/home/extend/lijiansong/work-space/anaconda2/envs/intel-caffe/lib/python3.6/site-packages/singa/layer.py",
line 45, in wrapper
   >     'initialize function expects PlaceHolders or Tensors')
   > AssertionError: initialize function expects PlaceHolders or Tensors
   > ```
   > 
   > Something maybe wrong with the layer initialization?
   > 
   > singa version: 3100(the latest build from the source code of master branch)
   > Python version: 3.5.2
   > ONNX version: 1.5.0
   
   Hi, @lijiansong , I cannot reproduce the error, I can see another error like this:
   ```
   WARNING: Logging before InitGoogleLogging() is written to STDERR
   W1013 10:55:05.660770  6279 convolution.cc:560] The required memory for workspace (2333081604)
is larger than the expected Bytes (1073741824)
   F1013 10:55:05.660809  6279 device.cc:88] Check failed: size >= 0 (-1961885692 vs. 0)
size is negative, could be caused by the type cast from size_t to int. In that case, the size
is too large.
   *** Check failure stack trace: ***
   Aborted (core dumped)
   ```
   
   The full log is:
   ```
   root@567b66a2525c:/singa# cd examples/onnx/
   root@567b66a2525c:/singa/examples/onnx# python3 training/train.py --model vgg16
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large
protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will
be halted for security reasons.  To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read
was 553437328
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large
protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will
be halted for security reasons.  To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read
was 553437328
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large
protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will
be halted for security reasons.  To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read
was 553437328
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large
protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will
be halted for security reasons.  To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read
was 553438994
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large
protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will
be halted for security reasons.  To increase the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
   [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read
was 553438996
   WARNING: Logging before InitGoogleLogging() is written to STDERR
   W1013 11:00:16.066620  6308 convolution.cc:560] The required memory for workspace (2333081604)
is larger than the expected Bytes (1073741824)
   F1013 11:00:16.066661  6308 device.cc:88] Check failed: size >= 0 (-1961885692 vs. 0)
size is negative, could be caused by the type cast from size_t to int. In that case, the size
is too large.
   *** Check failure stack trace: ***
   Aborted (core dumped)
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message