hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2774) Add counters to show number of key/values that have been sorted and merged in the maps and reduces
Date Wed, 19 Nov 2008 21:24:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649167#action_12649167

Chris Douglas commented on HADOOP-2774:

Sorry, I was unclear. The issue is not the volatility of the aggregator, but that it is static.
There are a number of contexts- JVM reuse, for one- where it simply will not work. Aggregating
disparate counters in a shared variable only to push its _deltas_ to another aggregator is
not an expected approach. A static counter 1) is already implemented in the Counter interface;
duplicating its functionality to feed it is suspect and 2) is almost certainly more difficult
to get right for all our use cases than some of the alternatives.

Sharad's proposal seems very reasonable. I'd suggest one variant: adding a Counter formal
to the IFile.Reader and IFile.Writer constructors. In the map, creating each Writer with a
counter to track each record hitting disk should be accurate. In the reduce, instead of incrementing
the counter as the segment is written to disk from the fetch and intermediate merges, updating
it as it is *read* from disk will yield the correct value at the end of the job. So for the
final merge into the reduce and the intermediate, on-disk merges, a counter will be provided.
This makes it unnecessary to transfer the record count to the reduce, lets the IFile format
remain exactly as it is, and should be fairly easy to implement.


> Add counters to show number of key/values that have been sorted and merged in the maps
and reduces
> --------------------------------------------------------------------------------------------------
>                 Key: HADOOP-2774
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2774
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.0
>         Attachments: HADOOP-2774.patch, HADOOP-2774.patch
> For each *pass* of the sort and merge, I would like a count of the number of records.
So for example, if the map output 100 records and they were sorted once, the counter would
be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level
merge, it may not be a multiple of the number of map output records. This would let the users
easily see if they have values like io.sort.mb or io.sort.factor set too low.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message