hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4882) Error in estimating the length of the output file in Spill Phase
Date Sat, 26 Jan 2013 12:39:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13563472#comment-13563472
] 

Hadoop QA commented on MAPREDUCE-4882:
--------------------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12566626/MAPREDUCE-4882.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3281//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3281//console

This message is automatically generated.
                
> Error in estimating the length of the output file in Spill Phase
> ----------------------------------------------------------------
>
>                 Key: MAPREDUCE-4882
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4882
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 1.0.3
>         Environment: Any Environment
>            Reporter: Lijie Xu
>            Assignee: Jerry Chen
>              Labels: patch
>         Attachments: MAPREDUCE-4882.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The sortAndSpill() method in MapTask.java has an error in estimating the length of the
output file. 
> The "long size" should be "(bufvoid - bufstart) + bufend" not "(bufvoid - bufend) + bufstart"
when "bufend < bufstart".
> Here is the original code in MapTask.java.
>  private void sortAndSpill() throws IOException, ClassNotFoundException,
>                                        InterruptedException {
>       //approximate the length of the output file to be the length of the
>       //buffer + header lengths for the partitions
>       long size = (bufend >= bufstart
>           ? bufend - bufstart
>           : (bufvoid - bufend) + bufstart) +
>                   partitions * APPROX_HEADER_LENGTH;
>       FSDataOutputStream out = null;
> ------------------------------------------------------------------------------
> I had a test on "TeraSort". A snippet from mapper's log is as follows:
> MapTask: Spilling map output: record full = true
> MapTask: bufstart = 157286200; bufend = 10485460; bufvoid = 199229440
> MapTask: kvstart = 262142; kvend = 131069; length = 655360
> MapTask: Finished spill 3
> In this occasioin, Spill Bytes should be (199229440 - 157286200) + 10485460 = 52428700
(52 MB) because the number of spilled records is 524287 and each record costs 100B.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message