hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Li (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (HADOOP-13849) Bzip2 java-builtin and system-native have almost the same compress speed
Date Thu, 01 Dec 2016 06:01:58 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tao Li updated HADOOP-13849:
----------------------------
    Comment: was deleted

(was: I)

> Bzip2 java-builtin and system-native have almost the same compress speed
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-13849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13849
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: common
>    Affects Versions: 2.6.0
>         Environment: os version: redhat6
> hadoop version: 2.6.0
> native bzip2 version: bzip2-devel-1.0.5-7.el6_0.x86_64
>            Reporter: Tao Li
>
> I tested bzip2 java-builtin and system-native compression, and I found the compress speed
is almost the same. (I think the system-native should have better compress speed than java-builtin)
> My test case:
> 1. input file: 2.7GB text file without compression
> 2. after bzip2 java-builtin compress: 457MB, 12min 4sec
> 3. after bzip2 system-native compress: 457MB, 12min 19sec
> My MapReduce Config:
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false");
> conf.set("mapreduce.output.fileoutputformat.compress", "true");
> conf.set("mapreduce.output.fileoutputformat.compress.type", "BLOCK");
> conf.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.BZip2Codec");
> conf.set("io.compression.codec.bzip2.library", "java-builtin"); // for java-builtin
> conf.set("io.compression.codec.bzip2.library", "system-native"); // for system-native
> And I am sure I have enable the bzip2 native, the output of command "hadoop checknative
-a" is as follows:
> Native library checking:
> hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
> zlib:    true /lib64/libz.so.1
> snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
> lz4:     true revision:99
> bzip2:   true /lib64/libbz2.so.1
> openssl: true /usr/lib64/libcrypto.so



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message