arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhavitvya Malik <bhavitvya.ma...@gmail.com>
Subject Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Date Thu, 11 Feb 2021 16:52:36 GMT
Sure, here it is:

data = np.zeros((10,8), dtype=np.uint8)
out = pa.array(list(data))
out.type  # ListType(list<item: uint8>)

data = np.zeros((3,4,6), dtype=np.uint8)
out = pa.array(list(data))  # Throws error ArrowInvalid: Can only
convert 1-dimensional array values

Even though it's working on 2D numpy arrays perfectly, it doesn't work on
N-Dimensional numpy arrays (where N > 2). Why is it so?

On Thu, 11 Feb 2021 at 21:18, Wes McKinney <wesmckinn@gmail.com> wrote:

> Can you provide more detail about what you are trying? You've showed
> some exception here but haven't showed the exact code that results in
> those exceptions
>
> On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik
> <bhavitvya.malik@gmail.com> wrote:
> >
> > Hi,
> > It's a follow up question for #9462. Rewriting the issue here:
> >
> >> I came to know that pyarrow has this limitation of not storing
> N-dimensional array. After looking into this issue, I decided to represent
> a N-dimensional array as a list of arrays i.e.
> >> data = np.zeros((5, 3), dtype=np.uint8)
> >> data = list(data)
> >> inorder to preserve the dtype but when it comes to typecasting and
> writing it into array (from list) pyarrow.array(data, type=type) it gives
> the following error:
> >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type
> numpy.ndarray: tried to convert to int
> >> Is there any way to avoid this issue? I just want to preserve the dtype
> from numpy array before converting it to list so that while writing it to
> pyarrow array format I can recognise its dtype and subsequently write it in
> that numpy dtype format.
> >
> >
> > I tried it with a 3D numpy array and it gave me this error even though
> it's working fine with 2D numpy arrays. Can you please look into this?
> >
> > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
> >
> >
> > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0
> too
> >
> >
> >
> > Thanks,
> > Bhavitvya
>

Mime
View raw message