Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <324220810.1227184247990.JavaMail.jira@brutus>
Date: Thu, 20 Nov 2008 04:30:47 -0800 (PST)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-2774) Add counters to show number of
 key/values that have been sorted and merged in the maps and reduces
In-Reply-To: <2132140.1201939988789.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649354#action_12649354 ] 

Devaraj Das commented on HADOOP-2774:
-------------------------------------

I was opposing the introduction of the mapred.Counters in IFile/Merger for the reason that IFile/Merger classes deal today with IO/merge only, and could very well be moved to the org.apache.hadoop.core package as part of the package split work. But this may not be preferred after all since we are also developing a more reusable file format TFile.
So I am okay either way I guess..

> Add counters to show number of key/values that have been sorted and merged in the maps and reduces
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2774
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2774
>             Project: Hadoop Core
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-2774.patch, HADOOP-2774.patch
>
>
> For each *pass* of the sort and merge, I would like a count of the number of records. So for example, if the map output 100 records and they were sorted once, the counter would be 100. If it spilled twice and was merged together, it would be 200. Clearly in a multi-level merge, it may not be a multiple of the number of map output records. This would let the users easily see if they have values like io.sort.mb or io.sort.factor set too low.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.