hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6647) MR usage counters use the resources requested instead of the resources allocated
Date Mon, 26 Jun 2017 21:54:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063865#comment-16063865
] 

Vrushali C commented on MAPREDUCE-6647:
---------------------------------------

Hi [~haibo.chen] [~rkanter]

Although it's technically a correct use case that we need a  counter that reports what the
actual usage is, this update will break the meaning for downstream apps which have been using
these counter to track their usage. They could now suddenly see a "drop" in their usage of
memory or vcores or both. 

Also, how do you envision tracking the resource allocation for these jobs? Is there another
counter that tells us how much memory was reserved for this job on the cluster (which is what
mbmillis meant before this patch)? Similarly for vcore-millis? Because that memory was not
available for scheduling other containers. 



> MR usage counters use the resources requested instead of the resources allocated
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6647
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>             Fix For: 2.9.0, 3.0.0-alpha1
>
>         Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, mapreduce6647.003.patch,
mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the resources
requested instead of the resources allocated. The scheduler increment-allocation-mb configs
could lead to these values not being the same. We could change the counters to use the allocated
resources in order to account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>       TaskAttemptImpl taskAttempt) {
>      /***omitted**/
>     long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
>     int mbRequired =
>         taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
>     int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);
>     int minSlotMemSize = taskAttempt.conf.getInt(
>       YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>       YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
>     int simSlotsRequired =
>         minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
>             / minSlotMemSize);
>     if (taskType == TaskType.MAP) {
>       jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
>       jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>       jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
>       jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
>     } else {
>       jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
>       jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
>       jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
>       jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message