hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4718) incrementing counters should not be used for triggering record skipping
Date Mon, 24 Nov 2008 21:57:44 GMT
incrementing counters should not be used for triggering record skipping
-----------------------------------------------------------------------

                 Key: HADOOP-4718
                 URL: https://issues.apache.org/jira/browse/HADOOP-4718
             Project: Hadoop Core
          Issue Type: Bug
          Components: mapred
            Reporter: Owen O'Malley


The following code is really problematic:

{code}
public void incrCounter(String group, String counter, long amount) {
  if (counters != null) {
    counters.incrCounter(group, counter, amount);
  }
  if(skipping && SkipBadRecords.COUNTER_GROUP.equals(group) && (
     SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS.equals(counter) ||
     SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS.equals(counter))) {
     //if application reports the processed records, move the 
     //currentRecStartIndex to the next.
     //currentRecStartIndex is the start index which has not yet been 
     //finished and is still in task's stomach.
     for(int i=0;i<amount;i++) {
        currentRecStartIndex = currentRecIndexIterator.next();
     }
   ...
}
{code}

In particular, if the user updates a counter with the wrong name, bad things will presumably
happen...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message