Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <478290520.1227129884398.JavaMail.jira@brutus>
Date: Wed, 19 Nov 2008 13:24:44 -0800 (PST)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-2774) Add counters to show number of
 key/values that have been sorted and merged in the maps and reduces
In-Reply-To: <2132140.1201939988789.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649167#action_12649167 ] 

Chris Douglas commented on HADOOP-2774:
---------------------------------------

Sorry, I was unclear. The issue is not the volatility of the aggregator, but that it is static. There are a number of contexts- JVM reuse, for one- where it simply will not work. Aggregating disparate counters in a shared variable only to push its _deltas_ to another aggregator is not an expected approach. A static counter 1) is already implemented in the Counter interface; duplicating its functionality to feed it is suspect and 2) is almost certainly more difficult to get right for all our use cases than some of the alternatives.

Sharad's proposal seems very reasonable. I'd suggest one variant: adding a Counter formal to the IFile.Reader and IFile.Writer constructors. In the map, creating each Writer with a counter to track each record hitting disk should be accurate. In the reduce, instead of incrementing the counter as the segment is written to disk from the fetch and intermediate merges, updating it as it is *read* from disk will yield the correct value at the end of the job. So for the final merge into the reduce and the intermediate, on-disk merges, a counter will be provided. This makes it unnecessary to transfer the record count to the reduce, lets the IFile format remain exactly as it is, and should be fairly easy to implement.

Thoughts?

> Add counters to show number of key/values that have been sorted and merged in the maps and reduces
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2774
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2774
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-2774.patch, HADOOP-2774.patch
>
>
> For each *pass* of the sort and merge, I would like a count of the number of records. So for example, if the map output 100 records and they were sorted once, the counter would be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level merge, it may not be a multiple of the number of map output records. This would let the users easily see if they have values like io.sort.mb or io.sort.factor set too low.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.