arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: pandas and Apache Arrow in context
Date Mon, 22 Feb 2016 23:14:24 GMT
Put that answer on the front page of the web site.

Well said.



On Mon, Feb 22, 2016 at 2:05 PM, Wes McKinney <wes@cloudera.com> wrote:

> hi Stuart,
>
> Currently pandas and NumPy only support flat, non-nested data. Nested
> data includes column value types including arrays, structs, maps, and
> unions. This enables you to analyze JSON-like data natively in-memory
> without pre-flattening or normalization.
>
> There's also an open question about how to handle nested data results
> from SQL engines like Spark SQL, Drill, and Impala, since there are
> currently native / C-level data structures (outside of pure Python
> data structures) in wide use to place the data arriving via RPC. Arrow
> serves to fill this need.
>
> - Wes
>
> On Mon, Feb 22, 2016 at 1:51 PM, Stuart Axelbrooke
> <stuart@axelbrooke.com> wrote:
> > Hey Wes,
> >
> > Very exciting to see things moving along on the Python front.  As you
> state
> > in your post, fast, ubiquitous columnar data will be a great foundation,
> > especially for more modern data processing and ETL tools.  Though I am a
> > bit curious what you mean by nested columnar data...
> >
> > Thanks,
> > Stuart
> >
> > On Mon, Feb 22, 2016 at 10:25 AM Wes McKinney <wes@cloudera.com> wrote:
> >
> >> hi all,
> >>
> >> I did a little bit of analysis of the costs of serialization
> bottlenecks in
> >> data access for Python pandas users and how (at a high level, no perf
> >> numbers yet!) Apache Arrow will help:
> >>
> >> http://wesmckinney.com/blog/pandas-and-apache-arrow/
> >>
> >> Feedback and comments welcome.
> >>
> >> cheers,
> >> Wes
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message