hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
Date Thu, 23 Jun 2011 01:22:47 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053586#comment-13053586
] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

@Jake 

snappy-java does not load vanilla snappy SO, it loads snappy-java SO which is a combination
of snappy SO + the JNI bindings for Java. 

Assuming that snappy-java splits the snappy & snappy-java native code in 2 SOs as I suggested
before in the snappy-java alias (and Taro is not convinced of that approach, see the end of
his comment) ... snappy SO would be loaded from the system but snappy-java SO would still
be loaded from the JAR. This would mean that not avail dependencies could still happen for
the snappy-java SO. And, if you want to have a snappy-java SO build for your OS, you'd have
build it yourself but still consume the JAR that comes an external dependency.

IMO this is a big NO NO. I rather have some extra setup work until snappy SO as commonly available
with the OSes.


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt,
v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt,
v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy).
This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression,
or compatibility with any other compression library; instead, it aims for very high speeds
and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is
an order of magnitude faster for most inputs, but the resulting compressed files are anywhere
from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses
at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message