arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng, Kai" <>
Subject RE: Understanding "shared" memory implications
Date Fri, 18 Mar 2016 02:01:49 GMT
Hi Wes,

Thanks for the helpful clarifying and am glad to know you're also well moving on the project.
Yes I will look closely to the project, raising my specific comments in the mailing list.


-----Original Message-----
From: Wes McKinney [] 
Sent: Friday, March 18, 2016 6:51 AM
Subject: Re: Understanding "shared" memory implications

hi Kai,

This sounds like it might merit a separate thread to discuss the growth of Arrow as a modular
ecosystem of libraries in different programming languages and related tools (we've discussed
shared memory data access and metadata representation, but not questions around ownership
and management of shared memory resources, which inevitably will come up). Composability around
the shared memory layout is an extremely powerful concept, just as zero-copy compatibility
with BLAS / LAPACK / Intel MKL is very important for high performance linear algebra in scientific
computing use cases.

(Note that I recently have been working to revive Parquet C++ as a standalone library and
have become a committer on that project, but my first commit was only in January. More specific
comments there might be valuable on the Parquet mailing list.)

- Wes

On Thu, Mar 17, 2016 at 3:44 PM, Zheng, Kai <> wrote:
> Sounds good to have all these compatible, modular goals and changes, for Apache Arrow,
in the early stage.
> On the other hand, not being afraid of changes keep evolving towards the core goals and
the well-defined initiatives, which is also important. Parquet is relevant and also a good
example. IMO, it's simply poorly organized. Yes it's all about history, but as a still young
project, it looks like to me it just stops evolving. Will Arrow be another Parquet?
> Regards,
> Kai
> -----Original Message-----
> From: Wes McKinney []
> Sent: Thursday, March 17, 2016 6:03 AM
> To:
> Subject: Re: Understanding "shared" memory implications
> On Wed, Mar 16, 2016 at 2:33 PM, Jacques Nadeau <> wrote:
>> For Arrow, let's make sure that we do our best to accomplish both (1) 
>> and (2). They seem like entirely compatible goals.
> For my part on the C++ side, I plan to proceed with a hub-and-spoke 
> model. A minimal small core library with "leaf" shared libraries (for
> example: Parquet read/write adapter) that you can opt-in to building.
> This will add some extra linker configuration complexity for downstream users but to
the benefit of a less monolithic library stack (which I don't think anyone wants).
View raw message