hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1061) Gridmix unit test should validate input/output bytes
Date Sat, 10 Oct 2009 06:53:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764309#action_12764309
] 

Chris Douglas commented on MAPREDUCE-1061:
------------------------------------------

Thanks for the review

bq. why do we need to set the actual extra bytes and records proportional to nMaps and nReds

If the spec expects 0 bytes/records, then the necessary spec data for each reduce needs to
be forgiven. The amount of extra data will be proportional to the number of maps/reduces.

However, this is adjacent to some sloppiness in the map output, where the spec data is not
written as part of the output, but rather as overhead. While the special case will still exist,
right now it's the case for all jobs. Since the test still needs to tolerate the 0 cases,
I was planning to tighten up the shuffle in a separate issue.

> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
>                 Key: MAPREDUCE-1061
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
>             Project: Hadoop Map/Reduce
>          Issue Type: Test
>    Affects Versions: 0.21.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>             Fix For: 0.21.0
>
>         Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been
run. The test should validate the I/O parameters it claims to satisfy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message