arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
Date Thu, 11 Feb 2021 17:39:54 GMT
We simply don't have conversions nor type inference implemented for
inner elements with dimension greater than 1. You're welcome to
propose this as a new feature / enhancement by opening a Jira issue.

On Thu, Feb 11, 2021 at 10:53 AM Bhavitvya Malik
<bhavitvya.malik@gmail.com> wrote:
>
> Sure, here it is:
>
> data = np.zeros((10,8), dtype=np.uint8)
> out = pa.array(list(data))
> out.type  # ListType(list<item: uint8>)
>
> data = np.zeros((3,4,6), dtype=np.uint8)
> out = pa.array(list(data))  # Throws error ArrowInvalid: Can only convert 1-dimensional
array values
>
> Even though it's working on 2D numpy arrays perfectly, it doesn't work on N-Dimensional
numpy arrays (where N > 2). Why is it so?
>
>
> On Thu, 11 Feb 2021 at 21:18, Wes McKinney <wesmckinn@gmail.com> wrote:
>>
>> Can you provide more detail about what you are trying? You've showed
>> some exception here but haven't showed the exact code that results in
>> those exceptions
>>
>> On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik
>> <bhavitvya.malik@gmail.com> wrote:
>> >
>> > Hi,
>> > It's a follow up question for #9462. Rewriting the issue here:
>> >
>> >> I came to know that pyarrow has this limitation of not storing N-dimensional
array. After looking into this issue, I decided to represent a N-dimensional array as a list
of arrays i.e.
>> >> data = np.zeros((5, 3), dtype=np.uint8)
>> >> data = list(data)
>> >> inorder to preserve the dtype but when it comes to typecasting and writing
it into array (from list) pyarrow.array(data, type=type) it gives the following error:
>> >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type numpy.ndarray:
tried to convert to int
>> >> Is there any way to avoid this issue? I just want to preserve the dtype
from numpy array before converting it to list so that while writing it to pyarrow array format
I can recognise its dtype and subsequently write it in that numpy dtype format.
>> >
>> >
>> > I tried it with a 3D numpy array and it gave me this error even though it's
working fine with 2D numpy arrays. Can you please look into this?
>> >
>> > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
>> >
>> >
>> > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 too
>> >
>> >
>> >
>> > Thanks,
>> > Bhavitvya

Mime
View raw message