hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
Date Tue, 13 Aug 2013 22:10:48 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13738908#comment-13738908

Jason Lowe commented on MAPREDUCE-5311:

bq. Defining SLOTS_MILLIS to be the same as CONTAINER_MILLIS will break compatibility for
the minority of clusters that both used the mapred.cluster.map.memory.mb property in MR1 and
chose to configure memory in MR2 in a way that exactly lines up with slots in MR1 (i.e. setting
yarn.scheduler.minimum-allocation-mb to exactly the same as mapred.cluster.map.memory.mb,
which is often not the right choice).

It will also break compatibility for those on 0.23 where SLOT_MILLIS is currently based on
the minimum allocation.  Changing the units from slots to containers for those cases will
significantly lower the reported value and could be misinterpreted as lower cluster utilization.
 That's why I'm not in favor of changing the SLOT_MILLIS units but still calling it SLOT_MILLIS.
 I'd rather the counter disappear than have it be a completely wrong value.  As Arun mentioned,
there are in-house tools to compute cluster utilization based on these metrics.

I agree that supporting SLOT_MILLIS long-term is problematic due to the removal of the minimum
or slot concept, hence my earlier suggestion for something more supportable like MEM_MILLIS
or CPU_MILLIS.  For backwards compatibility, we could consider doing something along the lines
of what it's already doing today, i.e.: having the AM read a config and computing the "slot"
size from there.  That way admins can configure whatever is appropriate for a "slot" for their
cluster to keep legacy tools reporting something appropriate when they try to look for SLOT_MILLIS.
 If the property is not configured then arguably the AM should not report a SLOT_MILLIS job
counter since it doesn't know how big a slot should be.
> Replace SLOTS_MILLIS counters with MEM_MILLIS
> ---------------------------------------------
>                 Key: MAPREDUCE-5311
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.0.4-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Sandy Ryza
>            Priority: Blocker
>             Fix For: 2.1.0-beta
>         Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch
> Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related
logic and just leave the counter constant for backwards compatibility and deprecate the counter

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message