hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-491) RAgzip: multiple map tasks for a large gzipped file
Date Wed, 07 Apr 2010 09:26:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854400#action_12854400
] 

Hadoop QA commented on MAPREDUCE-491:
-------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414707/MAPREDUCE-491.patch
  against trunk revision 931358.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/97/console

This message is automatically generated.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-491
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-491
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-4652-v2.patch, HADOOP-4652-v3.patch, HADOOP-4652.path, MAPREDUCE-491.patch
>
>
> Currently, the hadoop processes gzipped files with only one map.
> We have made a patch that enables multiple map tasks for one large gzipped file. We call
the patch RAgzip.
> To process multiple map tasks for gzipped file, you may use RAgzip by just changing InputFormat
to RAGZIPInputFormat.
> The option used in RAGZIPInputFormat can be found at the javadoc of RAGZIPInputFormat
part.
> RAgzip uses zlib's inflatePrime function which supports random access on a gzipped file.

> Since the inflatePrime is supported from the version of 1.2.2.4, it requires zlib 1.2.2.4
or higher. (We tested on zlib 1.2.3)
> RAgzip requires the preprocessing step that creates an access point (.ap) file, which
is like the index of the gzipped file chunks. 
> The access point(.ap) file is located in same path of the gzipped file.
> If there is a "/user/hadoop/test.gz", the .ap file is created with "/user/hadoop/test.gz.ap".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message