arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <>
Subject Re: IO considerations for PyArrow
Date Fri, 17 Jun 2016 17:22:36 GMT
hi Uwe,

Absent an obvious ASF project to move the code to (I'm intrigued by
the idea of an "Apache C++ common" project, much in the way that Boost
became a sprawling library ecosystem of sorts), there may not be a
clear incentive the split the code out if Arrow is the primary
consumer. Particularly since a major goal of Arrow early on is to
facilitate faster IO between systems (even if they don't use the Arrow
data structures as their primary ones).

I'm working on a patch for talking to HDFS (with libhdfs) -- I figure
we'll start there and see how things go.

- Wes

On Fri, Jun 17, 2016 at 7:01 AM, Uwe Korn <> wrote:
> Hello Wes,
> the concept sounds sensible and really useful.
> Probably the implementation will reside in the beginning fully inside of
> Arrow but do you plan to split it up into a separate package later on?
> Cheers
> Uwe

View raw message