hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Carlini (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo algorithm
Date Mon, 26 Jul 2010 19:32:25 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment: hadoop-6349-2.patch

Attached an update patch.

Fixed the checksum mismatch. It was possible for the decompressor to run out of input after
reading the header bytes but not notice if the block ID was 1. So if there were fewer than
26 bytes in the input (but more than 16) and the byte ID was 1 then it wouldn't notice and
just use whatever happened to be in the buffer at the time.

Fixed a bug in the decompressor where it would incorrectly indicate it was finished if at
the end of decompressing a block there was no more input left to decompress and decompress()
was then called again (TestCodec seed 1333275328, 2011623221, -1402938700 or -1990280158;
generate 50,000 records). Actually, the decompressor never returns finished now. This is because
the only time the decompressor should return true is if it somehow knows the end of the stream
has been reached and it doesn't, it just guesses that if it has read all the bytes it currently
has then it's done, which is not the case.

Implemented getRemaining().

Removed iOff from both the compressor and decompressor. It was initialized to zero from the
start and was only ever modified after that by setting it to 0.

Modified TestCodec to accept a seed as an argument.

Removed the rest of the carriage returns.

I will be adding a native version over the next few days and will upload that patch when it's
done.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, HADOOP-6349-TestFastLZCodec.patch,
HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good
(speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message