avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Russel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1414) Compression with C++ DataFile
Date Wed, 18 Dec 2013 23:28:07 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852297#comment-13852297

Daniel Russel commented on AVRO-1414:

I updated my patch to
- remove gzip
- set the parameters for zlib so as to not include the compression headers with zip (the java
impl the spec don't)
Hopefully this will make it compatible with the java implementation. It may also may make
sense to play with the window_bits or strategy (see http://www.boost.org/doc/libs/1_35_0/libs/iostreams/doc/classes/zlib.html),
but the (unspecified) defaults seem to be used in the java code.

The linked patch should now be clean on svn 1552153 <https://github.com/salilab/avrocpp/compare/d8afad009069f056168a6b10600fcf91a302b95a...compression>

> Compression with C++ DataFile
> -----------------------------
>                 Key: AVRO-1414
>                 URL: https://issues.apache.org/jira/browse/AVRO-1414
>             Project: Avro
>          Issue Type: Improvement
>          Components: c++
>            Reporter: Daniel Russel
>         Attachments: AVRO-1414.patch, patch
> There is no way to use compression with the C++ DataFileReader and C++ DataFileWriter,
from what I can tell. Adding compression of the written blocks using boost streams is relatively
straight forward and I can provide a patch if people are interested. 
> However, there are a couple caveats:
> - the windows builds of boost don't currently include zlib support (required for compression)
by default. You have to do extra work to get it.
> - I don't know if doing it that way is compatible with other avro implementations

This message was sent by Atlassian JIRA

View raw message