Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 26 Jun 2017 21:54:00 +0000 (UTC)
From: "Vrushali C (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12946321.1456960266000.109454.1498514040662@Atlassian.JIRA>
In-Reply-To: <JIRA.12946321.1456960266000@Atlassian.JIRA>
References: <JIRA.12946321.1456960266000@Atlassian.JIRA> <JIRA.12946321.1456960266821@jira-lw-us.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-6647) MR usage counters use the
 resources requested instead of the resources allocated
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 26 Jun 2017 21:54:16 -0000


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063865#comment-16063865 ] 

Vrushali C commented on MAPREDUCE-6647:
---------------------------------------

Hi [~haibo.chen] [~rkanter]

Although it's technically a correct use case that we need a  counter that reports what the actual usage is, this update will break the meaning for downstream apps which have been using these counter to track their usage. They could now suddenly see a "drop" in their usage of memory or vcores or both. 

Also, how do you envision tracking the resource allocation for these jobs? Is there another counter that tells us how much memory was reserved for this job on the cluster (which is what mbmillis meant before this patch)? Similarly for vcore-millis? Because that memory was not available for scheduling other containers. 


> MR usage counters use the resources requested instead of the resources allocated
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6647
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6647
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>             Fix For: 2.9.0, 3.0.0-alpha1
>
>         Attachments: mapreduce6647.001.patch, mapreduce6647.002.patch, mapreduce6647.003.patch, mapreduce6647.004.patch
>
>
> As can be seen in the following snippet, the MR counters for usage use the resources requested instead of the resources allocated. The scheduler increment-allocation-mb configs could lead to these values not being the same. We could change the counters to use the allocated resources in order to account for this.
> {code}
>   private static void updateMillisCounters(JobCounterUpdateEvent jce,
>       TaskAttemptImpl taskAttempt) {
>      /***omitted**/
>     long duration = (taskAttempt.getFinishTime() - taskAttempt.getLaunchTime());
>     int mbRequired =
>         taskAttempt.getMemoryRequired(taskAttempt.conf, taskType);
>     int vcoresRequired = taskAttempt.getCpuRequired(taskAttempt.conf, taskType);
>     int minSlotMemSize = taskAttempt.conf.getInt(
>       YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
>       YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
>     int simSlotsRequired =
>         minSlotMemSize == 0 ? 0 : (int) Math.ceil((float) mbRequired
>             / minSlotMemSize);
>     if (taskType == TaskType.MAP) {
>       jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_MAPS, simSlotsRequired * duration);
>       jce.addCounterUpdate(JobCounter.MB_MILLIS_MAPS, duration * mbRequired);
>       jce.addCounterUpdate(JobCounter.VCORES_MILLIS_MAPS, duration * vcoresRequired);
>       jce.addCounterUpdate(JobCounter.MILLIS_MAPS, duration);
>     } else {
>       jce.addCounterUpdate(JobCounter.SLOTS_MILLIS_REDUCES, simSlotsRequired * duration);
>       jce.addCounterUpdate(JobCounter.MB_MILLIS_REDUCES, duration * mbRequired);
>       jce.addCounterUpdate(JobCounter.VCORES_MILLIS_REDUCES, duration * vcoresRequired);
>       jce.addCounterUpdate(JobCounter.MILLIS_REDUCES, duration);
>     }
> {code}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org