arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe L. Korn (JIRA)" <>
Subject [jira] [Commented] (ARROW-300) [Format] Add buffer compression option to IPC file format
Date Tue, 15 Nov 2016 19:15:58 GMT


Uwe L. Korn commented on ARROW-300:

Given my latest (sadly internal) performance tests, I'm not so sure about the benefit of a
compressed arrow file format. For me the main distinction is that Parquet provides efficient
storage (with the tradeoff of not being to randomly access a single row) and Arrow random
access, both for columnar data.

The one point where I see an Arrow file format as beneficial is where you need random access
to its data but cannot load it fully into RAM but instead use a memory mapped file. If you
add compression (either column-wise or whole-file level), you cannot memorymap it anymore.

The only point where I can see that having columnar compression for Arrow batches is better
than on the whole file layer is that it actually produces better compression behaviour. This
means that doing compression on a per-column basis can be parallelised independently of the
underyling algorithm thus leading to better CPU usage. Furthermore the compression may be
better if done on a column level (with a sufficient number of rows) as the data inside a column
is very similar thus leading to smaller compression dictionaries and better compresssion ratios
at the end. Both things mentioned are just assumptions that should be tested before being

> [Format] Add buffer compression option to IPC file format
> ---------------------------------------------------------
>                 Key: ARROW-300
>                 URL:
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Format
>            Reporter: Wes McKinney
> It may be useful if data is to be sent over the wire to compress the data buffers themselves
as their being written in the file layout.
> I would propose that we keep this extremely simple with a global buffer compression setting
in the file Footer. Probably only two compressors worth supporting out of the box would be
zlib (higher compression ratios) and lz4 (better performance).
> What does everyone think?

This message was sent by Atlassian JIRA

View raw message