drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion
URL: https://github.com/apache/incubator-mxnet/issues/16960
## Description
Conversion fails on large matrices.
### Error Message
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line
146, in array
return _array(source_array, ctx=ctx, dtype=dtype)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
2505, in array
arr[:] = source_array
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
449, in __setitem__
self._set_nd_basic_indexing(key, value)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
715, in _set_nd_basic_indexing
self._sync_copyfrom(value)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line
881, in _sync_copyfrom
ctypes.c_size_t(source_array.size)))
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/base.py", line 253, in
check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:09:04] src/ndarray/ndarray_function.cc:51: Check failed: size
== to->Size() (-294967296 vs. 4000000000) : copying size mismatch, from: 18446744072529682432
bytes, to: 16000000000 bytes.
Stack trace:
[bt] (0) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2795cb)
[0x7efff4d2c5cb]
[bt] (1) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x259399b)
[0x7efff704699b]
[bt] (2) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void
const*, unsigned long) const+0x284) [0x7efff6fe5934]
[bt] (3) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b)
[0x7efff6d8ebcb]
[bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f0059bbacec]
[bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7f0059bba615]
[bt] (6) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2a0)
[0x7f0059dcd290]
[bt] (7) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9586)
[0x7f0059dc6586]
[bt] (8) /usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x90) [0x7f0061bce7e0]
### Steps to reproduce
T= nd.array(np.random.randn(5000000,800))
(Paste the commands you ran that produced the error.)
## What have you tried to solve it?
1. workaround: If you convert in smaller chunks and concatenate the ndarray to create the
final mxnet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
|