mxnet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-mxnet] drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion
Date Mon, 02 Dec 2019 19:58:03 GMT
drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion
URL: https://github.com/apache/incubator-mxnet/issues/16960
 
 
   ## Description
   Conversion fails on large matrices.
   
   ### Error Message
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line
146, in array
       return _array(source_array, ctx=ctx, dtype=dtype)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
2505, in array
       arr[:] = source_array
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
449, in __setitem__
       self._set_nd_basic_indexing(key, value)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
715, in _set_nd_basic_indexing
       self._sync_copyfrom(value)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
881, in _sync_copyfrom
       ctypes.c_size_t(source_array.size)))
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/base.py", line 253, in
check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [19:09:04] src/ndarray/ndarray_function.cc:51: Check failed: size
== to->Size() (-294967296 vs. 4000000000) : copying size mismatch, from: 18446744072529682432
bytes, to: 16000000000 bytes.
   Stack trace:
     [bt] (0) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2795cb)
[0x7efff4d2c5cb]
     [bt] (1) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x259399b)
[0x7efff704699b]
     [bt] (2) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void
const*, unsigned long) const+0x284) [0x7efff6fe5934]
     [bt] (3) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b)
[0x7efff6d8ebcb]
     [bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f0059bbacec]
     [bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7f0059bba615]
     [bt] (6) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2a0)
[0x7f0059dcd290]
     [bt] (7) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9586)
[0x7f0059dc6586]
     [bt] (8) /usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x90) [0x7f0061bce7e0]
   
   
   ### Steps to reproduce
   T= nd.array(np.random.randn(5000000,800))
   (Paste the commands you ran that produced the error.)
   
   ## What have you tried to solve it?
   
   1. workaround: If you convert in smaller chunks and concatenate the ndarray to create the
final mxnet
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message