hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5958) Wrong reduce task progress if map output is compressed
Date Sat, 05 Jul 2014 17:19:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052926#comment-14052926
] 

Hadoop QA commented on MAPREDUCE-5958:
--------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12654179/HADOOP-5958-v2.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified
tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4715//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4715//console

This message is automatically generated.

> Wrong reduce task progress if map output is compressed
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-5958
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5958
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.2.0, 2.3.0, 2.2.1, 2.4.0, 2.4.1
>            Reporter: Emilio Coppa
>            Priority: Minor
>              Labels: progress, reduce
>             Fix For: 2.4.1
>
>         Attachments: HADOOP-5958-v2.patch
>
>
> If the map output is compressed (_mapreduce.map.output.compress_ set to _true_) then
the reduce task progress may be highly underestimated.
> In the reduce phase (but also in the merge phase), the progress of a reduce task is computed
as the ratio between the number of processed bytes and the number of total bytes. But:
> - the number of total bytes is computed by summing up the uncompressed segment sizes
(_Merger.Segment.getRawDataLength()_)
> - the number of processed bytes is computed by exploiting the position of the current
_IFile.Reader_ (using _IFile.Reader.getPosition()_) but this may refer to the position in
the underlying on disk file (which may be compressed)
> Thus, if the map outputs are compressed then the progress may be underestimated (e.g.,
only 1 map output ondisk file, the compressed file is 25% of its original size, then the reduce
task progress during the reduce phase will range between 0 and 0.25 and then artificially
jump to 1.0).
> Attached there is a patch: the number of processed bytes is now computed by exploiting
_IFile.Reader.bytesRead_ (if the the reader is in memory, then _getPosition()_ already returns
exactly this field).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message