hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2774) Add counters to show number of key/values that have been sorted and merged in the maps and reduces
Date Fri, 21 Nov 2008 08:58:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649627#action_12649627
] 

Chris Douglas commented on HADOOP-2774:
---------------------------------------

OK. Talked with Jothi offline. The second counter is handling the _intermediate merge_, which
creates its own Writer independent of the counter for records emitted from the RawKeyValueIterator.
Specifying how/whether to count these records requires more than the single, Counter parameter
to the merge.

I'm +1 on the approach as described.

> Add counters to show number of key/values that have been sorted and merged in the maps
and reduces
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2774
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2774
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-2774.patch, HADOOP-2774.patch
>
>
> For each *pass* of the sort and merge, I would like a count of the number of records.
So for example, if the map output 100 records and they were sorted once, the counter would
be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level
merge, it may not be a multiple of the number of map output records. This would let the users
easily see if they have values like io.sort.mb or io.sort.factor set too low.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message