arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <>
Subject Re: Understanding "shared" memory implications
Date Thu, 17 Mar 2016 22:51:00 GMT
hi Kai,

This sounds like it might merit a separate thread to discuss the
growth of Arrow as a modular ecosystem of libraries in different
programming languages and related tools (we've discussed shared memory
data access and metadata representation, but not questions around
ownership and management of shared memory resources, which inevitably
will come up). Composability around the shared memory layout is an
extremely powerful concept, just as zero-copy compatibility with BLAS
/ LAPACK / Intel MKL is very important for high performance linear
algebra in scientific computing use cases.

(Note that I recently have been working to revive Parquet C++ as a
standalone library and have become a committer on that project, but my
first commit was only in January. More specific comments there might
be valuable on the Parquet mailing list.)

- Wes

On Thu, Mar 17, 2016 at 3:44 PM, Zheng, Kai <> wrote:
> Sounds good to have all these compatible, modular goals and changes, for Apache Arrow,
in the early stage.
> On the other hand, not being afraid of changes keep evolving towards the core goals and
the well-defined initiatives, which is also important. Parquet is relevant and also a good
example. IMO, it's simply poorly organized. Yes it's all about history, but as a still young
project, it looks like to me it just stops evolving. Will Arrow be another Parquet?
> Regards,
> Kai
> -----Original Message-----
> From: Wes McKinney []
> Sent: Thursday, March 17, 2016 6:03 AM
> To:
> Subject: Re: Understanding "shared" memory implications
> On Wed, Mar 16, 2016 at 2:33 PM, Jacques Nadeau <> wrote:
>> For Arrow, let's make sure that we do our best to accomplish both (1)
>> and (2). They seem like entirely compatible goals.
> For my part on the C++ side, I plan to proceed with a hub-and-spoke model. A minimal
small core library with "leaf" shared libraries (for
> example: Parquet read/write adapter) that you can opt-in to building.
> This will add some extra linker configuration complexity for downstream users but to
the benefit of a less monolithic library stack (which I don't think anyone wants).

View raw message