hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
Date Wed, 02 Mar 2016 23:11:18 GMT
Haibo Chen created MAPREDUCE-6647:

             Summary: MR usage counters use the resources requested instead of the resources
                 Key: MAPREDUCE-6647
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
             Project: Hadoop Map/Reduce
          Issue Type: Bug
            Reporter: Haibo Chen
            Assignee: Haibo Chen

As can be seen in the following snippet, the MR counters for usage use the resources requested
instead of the resources allocated. The scheduler increment-allocation-mb configs could lead
to these values not being the same. We could change the counters to use the allocated resources
in order to account for this.

  private static void updateMillisCounters(JobCounterUpdateEvent jce,
      TaskAttemptImpl taskAttempt) {
  long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
    int mbRequired =
        taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
    int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);

    int minSlotMemSize = taskAttempt.conf.getInt(

    int simSlotsRequired =
        minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
            / minSlotMemSize);

    if (taskType == TaskType.MAP) {
      jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
      jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
      jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
      jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
    } else {
      jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
      jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
      jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
      jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);

This message was sent by Atlassian JIRA

View raw message