arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cindy McMullen <cmcmul...@twitter.com>
Subject Re: Avro and Thrift converters
Date Wed, 20 May 2020 19:36:23 GMT
Hi, Micah -

I wasn't aware that the Avro converter already existed in Java, since I
couldn't find any Arrow docs on it. I was going by the Arrow/JIRA release
tag.  Are you saying that the Avro -> Arrow converter is already available
in release 0.17.1?

As for use cases: we're trying to move away from Thrift in parts of our ML
stack.  We need to support wide, row-based data with schema support, so
probably need to convert Thrift to Avro.  However, we'd love to use Arrow
*between* components (Spark, TensorFlow, scikit-learn), but it's likely our
data will originate in Avro and/or Thrift.

Thanks -

-- Cindy

On Wed, May 20, 2020 at 1:14 PM Micah Kornfield <emkornfield@gmail.com>
wrote:

> The  avro to arrow converter in c++/python will not be done anytime soon
> unless someone else takes it up (one exists in Java).  It has been on my
> low priority backlog for a while but I haven't had time to get to it.  We
> should remove a specific release tag from it.
>
> As far as I know there are no plans for thrift or other formats at this
> point.
>
> May I ask what your use case is?
>
> Thanks,
> Micah
>
> On Wednesday, May 20, 2020, Cindy McMullen <cmcmullen@twitter.com> wrote:
>
>> I see that the Avro converter is planned for Arrow 1.0.0.  Any ideas
>> about when that release might be?
>>
>> Any plans for a Thrift -> Avro converter?
>>
>

Mime
View raw message