arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vibhatha Abeykoon <vibha...@gmail.com>
Subject Re: PyArrow Compute is_null Usage
Date Wed, 18 Nov 2020 19:48:59 GMT
I understand your point. Even if null elements are present, it would
consider as not_null. I assumed this operator would be similar in function
to Pandas isnull.

import pandas as pd
pdf = pd.DataFrame([[1, 2, 3, 4, None], [11, 12, 13, None, 15]])
pdf.isnull()

For instance in this one, element-wise check is done.


With Regards,
Vibhatha Abeykoon


On Wed, Nov 18, 2020 at 2:28 PM Micah Kornfield <emkornfield@gmail.com>
wrote:

> It looks right.  Could you clarify why you think it might not be expected
> behavior?  The arrays that are being constructed are of two different
> types.  One is an integer array (first example).   The rest on List<Int>.
> None of the lists in the examples are null (but they do contain null
> elements).
>
> On Wed, Nov 18, 2020 at 11:19 AM Vibhatha Abeykoon <vibhatha@gmail.com>
> wrote:
>
>> Hello,
>>
>> I am looking into the is_null compute and observed the following.
>>
>> import pyarrow as pa
>> arw_ar = pa.array([1, 2, 3, 4, None])
>> arw_ar_1 = pa.array([[1, 2, 3, 4, None], [11, 12, 13, None, 15]])
>> arw_ar_2 = pa.array([[1, 2, 3, 4], [1, 2, 3, 4]])
>> arw_ar_3 = pa.array([[None, None, None, None, None], [11, 12, 13, None,
>> 15]])
>>
>> # Case 1 with random None value in a 1D array
>>
>> print(arw_ar.is_null())
>>
>> # [
>> # false,
>> # false,
>> # false,
>> # false,
>> # true
>> # ]
>>
>> # Case 2 with random None value in a 2-D array
>>
>> print(arw_ar_1.is_null())
>>
>> # [
>> # false,
>> # false
>> # ]
>>
>> # Case 3 without random None value in a 2-D array
>> print(arw_ar_2.is_null())
>>
>> # [
>> # false,
>> # false
>> # ]
>>
>> # Case 4 with None value in a 2-D array
>> print(arw_ar_3.is_null())
>> # [
>> # false,
>> # false
>> # ]
>>
>> Is this an expected behavior?
>>
>>
>> With Regards,
>> Vibhatha Abeykoon,
>>
>>

Mime
View raw message