arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <>
Subject [DISCUSS] C++ code sharing amongst Apache {Arrow, Kudu, Impala, Parquet}
Date Sat, 25 Feb 2017 22:18:36 GMT
Dear Apache Kudu and Apache Impala (incubating) communities,

(I'm not sure the best way to have a cross-list discussion, so I
apologize if this does not work well)

On the recent Apache Parquet sync call, we discussed C++ code sharing
between the codebases in Apache Arrow and Apache Parquet, and
opportunities for more code sharing with Kudu and Impala as well.

As context

* We have an RC out for the 1.0.0 release of apache-parquet-cpp, the
first C++ release within Apache Parquet. I got involved with this
project a little over a year ago and was faced with the unpleasant
decision to copy and paste a significant amount of code out of
Impala's codebase to bootstrap the project.

* In parallel, we begin the Apache Arrow project, which is designed to
be a complementary library for file formats (like Parquet), storage
engines (like Kudu), and compute engines (like Impala and pandas).

* As Arrow and parquet-cpp matured, an increasing amount of code
overlap crept up surrounding buffer memory management and IO
interface. We recently decided in PARQUET-818
to remove some of the obvious code overlap in Parquet and make
libarrow.a/so a hard compile and link-time dependency for

* There is still quite a bit of code in parquet-cpp that would better
fit in Arrow: SIMD hash utilities, RLE encoding, dictionary encoding,
compression, bit utilities, and so forth. Much of this code originated
from Impala

This brings me to a next set of points:

* parquet-cpp contains quite a bit of code that was extracted from
Impala. This is mostly self-contained in

* My understanding is that Kudu extracted certain computational
utilities from Impala in its early days, but these tools have likely
diverged as the needs of the projects have evolved.

Since all of these projects are quite different in their end goals
(runtime systems vs. libraries), touching code that is tightly coupled
to either Kudu or Impala's runtimes is probably not worth discussing.
However, I think there is a strong basis for collaboration on
computational utilities and vectorized array processing. Some obvious
areas that come to mind:

* SIMD utilities (for hashing or processing of preallocated contiguous memory)
* Array encoding utilities: RLE / Dictionary, etc.
* Bit manipulation (packing and unpacking, e.g. Daniel Lemire
contributed a patch to parquet-cpp around this)
* Date and time utilities
* Compression utilities

I hope the benefits are obvious: consolidating efforts on unit
testing, benchmarking, performance optimizations, continuous
integration, and platform compatibility.

Logistically speaking, one possible avenue might be to use Apache
Arrow as the place to assemble this code. Its thirdparty toolchain is
small, and it builds and installs fast. It is intended as a library to
have its headers used and linked against other applications. (As an
aside, I'm very interested in building optional support for Arrow
columnar messages into the kudu client).

The downside of code sharing, which may have prevented it so far, are
the logistics of coordinating ASF release cycles and keeping build
toolchains in sync. It's taken us the past year to stabilize the
design of Arrow for its intended use cases, so at this point if we
went down this road I would be OK with helping the community commit to
a regular release cadence that would be faster than Impala, Kudu, and
Parquet's respective release cadences. Since members of the Kudu and
Impala PMC are also on the Arrow PMC, I trust we would be able to
collaborate to each other's mutual benefit and success.

Note that Arrow does not throw C++ exceptions and similarly follows
Google C++ style guide to the same extent at Kudu and Impala.

If this is something that either the Kudu or Impala communities would
like to pursue in earnest, I would be happy to work with you on next
steps. I would suggest that we start with something small so that we
could address the necessary build toolchain changes, and develop a
workflow for moving around code and tests, a protocol for code reviews
(e.g. Gerrit), and coordinating ASF releases.

Let me know what you think.


View raw message