hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants
Date Sat, 03 Aug 2013 00:47:49 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728333#comment-13728333
] 

Sandy Ryza commented on MAPREDUCE-5311:
---------------------------------------

SLOTS_MILLIS_MAPS is a confusing property that I think we should remove if at all possible.
 Independent of YARN-1004, the current way of calculating it is a hack that neither adheres
to its behavior in MR1 nor its description.

In MR1, it was calculated as the sum of all map task's running times multiplied by the number
of slots each task took up.

The description, "Total time spent by all maps in occupied slots (ms)", with its unit being
ms and not slots-ms, reads to me as a sum of all map task's running times, independent of
the number of slots consumed by each. 

On most clusters, however, these were equivalent, because a map task would only take up more
than one slot if mapred.cluster.map.memory.mb were set to something other than its default,
-1.

The MR2 counter, which is calculated as the sum of all map task's running times multiplied
by the memory allocated by each one, divided by the minimum allocation, adheres to neither
of these.  The average MR1 user who did not set mapred.cluster.map.memory.mb will find, upon
upgrading the MR2, that SLOTS_MILLIS_MAPS functions differently than it did in MR1, as well
as be confused by YARN's claim that it gets rid of slots.

This is not an obscure feature.  It's one of the things that any person running a MapReduce
job will see, and I think keeping it in its current form will help to confuse those who are
trying to upgrade to MR2 and understand the new resource model. 
                
> Remove slot millis computation logic and deprecate counter constants
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5311
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.0.4-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Sandy Ryza
>         Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch
>
>
> Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related
logic and just leave the counter constant for backwards compatibility and deprecate the counter
constants.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message