hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-538) Implement a nio's 'direct buffer' based wrapper over zlib to improve performance of java.util.zip.{De|In}flater as a 'custom codec'
Date Thu, 28 Sep 2006 06:52:51 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-538?page=comments#action_12438331 ] 
            
Dawid Weiss commented on HADOOP-538:
------------------------------------

Doug is correct -- compiling native code is a pain and including it in the runtime is even
more pain. I have a project that uses JNI to bind to a shared library, which in turn calls
other libraries. Apart from setting up the build process in the ANT build file, which is itself
quite tricky -- the same set of tools (gcc) on Linux produces *.so, whereas on Cygwin it creates
*.dll files -- you need to set up java.library.path to point to your shared objects AND a
regular path (Windows) or LD_LIBRARY_PATH (Linux-ish) so that the system can find native dependencies.
In the end, it's a real pain to compile and maintain.

The suggestion to fallback to pure java is I think a good one. It allows you to start playing
with Hadoop on heterogenic environment and without much hassle and then tune it later (by
placing appropriate libraries wherever the system can find them).

> Implement a nio's 'direct buffer' based wrapper over zlib to improve performance of java.util.zip.{De|In}flater
as a 'custom codec'
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-538
>                 URL: http://issues.apache.org/jira/browse/HADOOP-538
>             Project: Hadoop
>          Issue Type: Improvement
>    Affects Versions: 0.6.1
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.7.0
>
>         Attachments: HADOOP-538.patch, HADOOP-538_benchmarks.tgz
>
>
> There has been more than one instance where java.util.zip's {De|In}flater classes perform
unreliably, a simple wrapper over zlib-1.2.3 (latest stable) using java.nio.ByteBuffer (i.e.
direct buffers) should go a long way in alleviating these woes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message