hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6067) native-task: spilled records counter is incorrect
Date Wed, 03 Sep 2014 21:13:56 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120469#comment-14120469

Todd Lipcon commented on MAPREDUCE-6067:

I also noticed this commented-out assertion:

//        assertEquals("Native Reduce reduce group counter should equal orignal reduce group
//            nativeReduceGroups.getValue(), normalReduceGroups.getValue());

Why is it commented out?

Perhaps we should re-enable assertions like this for all of the map counters.

> native-task: spilled records counter is incorrect
> -------------------------------------------------
>                 Key: MAPREDUCE-6067
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6067
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: task
>            Reporter: Todd Lipcon
>            Assignee: Binglin Chang
>         Attachments: MAPREDUCE-6067.v1.patch, native-counters.html, trunk-counters.html
> After running a terasort, I see the spilled records counter at 5028651606, which is about
half what I expected to see. Using the non-native collector I see the expected count of 10000000000.
It seems the correct number of records were indeed spilled, because the job's output record
count is correct.

This message was sent by Atlassian JIRA

View raw message