hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Kang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6683) the first optimization: ZlibCompressor does not fully utilize the buffer
Date Wed, 07 Apr 2010 03:32:33 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854316#action_12854316
] 

Xiao Kang commented on HADOOP-6683:
-----------------------------------

A comparision test was performed on a 1.8GB web log file. The result is as follows:

|| read file buffer size || write to compress stream buffer size || old time(secs) || new
time(secs) || decrease % ||
|65536|	100 |67|	49|	26.8%|
|65536|	200| 56.5|	46.5|	17.7%|
|65536|	400| 51.5|	45|	12.6%|
|65536|	800| 48.5|	44.5|	8.2%|
|65536|	1024|	46.8|	44.2|	9.8%|
|65536|	4096|	45|	43.5|	3.3%|
|65536|	65536|	44.6|	43.2|	3.1%|


Is there any standard benchmark for compression suitable for this case?

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message