hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6297) Hadoop's support for zlib library lacks support to perform flushes (Z_SYNC_FLUSH and Z_FULL_FLUSH)
Date Sun, 18 Oct 2009 08:52:31 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Douglas updated HADOOP-6297:
----------------------------------

    Affects Version/s:     (was: site)
               Status: Open  (was: Patch Available)

bq. It looks like the native library isn't getting initialized properly by the testing machine,
for some reason. I can run the test just fine locally using ant test.

The test should pass even if the native libraries aren't loaded, as some platforms (Windows,
MacOS) don't come with the native libs installed. The error in the unit test:
{noformat}
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.compress.zlib.ZlibCompressor.init(III)J
{noformat}
may be a problem with Hudson, but you may want to be certain your patch works on a clean checkout
of trunk.

bq. Also, I'm confused by the audit warning. What does that warning mean?

It means that the patch adds files without headers licensing them to Apache. TestZlib.java,
in this case.

Small nits on the testcase:
* Please use the JUnit4 conventions, rather than JUnit3; instead of extending TestCase, import
the org.junit classes and use annotations (see examples in src/test, e.g. TestCodec)
* This can simply call {{Assert.fail()}} rather than using the boolean, or even better, just
allow the exception to escape the method

> Hadoop's support for zlib library lacks support to perform flushes (Z_SYNC_FLUSH and
Z_FULL_FLUSH)
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6297
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Kevin J. Price
>            Priority: Minor
>         Attachments: zlibpatch.patch
>
>
> The zlib library supports the ability to perform two types of flushes when deflating
data. It can perform both a Z_SYNC_FLUSH, which forces all input to be written as output and
byte-aligned and resets the Huffman coding, and it also supports a Z_FULL_FLUSH, which does
the same thing but additionally resets the compression dictionary.  The Hadoop wrapper for
the zlib library does not support either of these two methods.
> Adding support should be fairly trivial.  An additional deflate method that takes a fourth
"flush" parameter, and a modification to the native c code to accept this fourth parameter
and pass it along to the zlib library.  I can submit a patch for this if desired.
> It should be noted that the native SUN Java API is likewise missing this functionality,
as has been noted for over a decade here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4206909

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message