hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daehyun Kim (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4652) RAgzip: multiple map tasks for a large gzipped file
Date Thu, 13 Nov 2008 10:01:44 GMT
RAgzip: multiple map tasks for a large gzipped file

                 Key: HADOOP-4652
                 URL: https://issues.apache.org/jira/browse/HADOOP-4652
             Project: Hadoop Core
          Issue Type: Improvement
          Components: io, mapred, native
    Affects Versions: 0.18.2
            Reporter: Daehyun Kim

Currently, the hadoop processes gzipped files with only one map.
We have made a patch that enables multiple map tasks for one large *gzipped* file. We call
the patch RAgzip.
To process multiple map tasks for gzipped file, you may use RAgzip by just changing InputFormat
to RAGZIPInputFormat.
The option used in RAGZIPInputFormat can be found at the javadoc of RAGZIPInputFormat part.

RAgzip uses zlib's inflatePrime function which supports random access on a gzipped file. 
Since the inflatePrime is supported from the version of, it requires zlib
or higher. (We tested on zlib 1.2.3)
RAgzip requires the preprocessing step that creates an access point (.ap) file, which is like
the index of the gzipped file chunks. 
The access point(.ap) file is located in same path of the gzipped file.
If there is a "/user/hadoop/test.gz", the .ap file is created with "/user/hadoop/test.gz.ap".

We made two patches. 
1. One makes changes in the source of the hadoop core. This is the main patch. 
If the zlib version of the hadoop cluster is greater than, you should use this patch.
2. On the other hand, if there is a computer with zlib version less than in hadoop
cluster, you should use the other patch. 
This patch uses static link library of the zlib. So if you compile this patch once at a computer
with zlib version greater than, RAgzip can be used in hadoop cluster even if a computer
with zlib version less than exists in the cluster.

As you know, second patch creates jar file(build/contrib/ragzip/hadoop-x.xx.x-dev-ragzip.jar)
as a result of its installation.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message