hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Li (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13849) Bzip2 java-builtin and system-native have almost the same compress speed
Date Wed, 30 Nov 2016 13:10:58 GMT
Tao Li created HADOOP-13849:

             Summary: Bzip2 java-builtin and system-native have almost the same compress speed
                 Key: HADOOP-13849
                 URL: https://issues.apache.org/jira/browse/HADOOP-13849
             Project: Hadoop Common
          Issue Type: Bug
          Components: common
    Affects Versions: 2.6.0
         Environment: os version: redhat6
hadoop version: 2.6.0
native bzip2 version: bzip2-devel-1.0.5-7.el6_0.x86_64

            Reporter: Tao Li

I tested bzip2 java-builtin and system-native compression, and I found the compress speed
is almost the same. (I think the system-native should have better compress speed than java-builtin)

My test case:
input: 2.7GB text file without compression
bzip2 java-builtin compress: 457MB, 12min 4sec
bzip2 system-native compress: 457MB, 12min 19sec

My MapReduce Config:
conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false");
conf.set("mapreduce.output.fileoutputformat.compress", "true");
conf.set("mapreduce.output.fileoutputformat.compress.type", "BLOCK");
conf.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.BZip2Codec");
conf.set("io.compression.codec.bzip2.library", "java-builtin"); // for java-builtin
conf.set("io.compression.codec.bzip2.library", "system-native"); // for system-native

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message