hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3255) MR2 needs a counter analogous to "slot seconds"
Date Tue, 25 Oct 2011 17:20:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135246#comment-13135246
] 

Todd Lipcon commented on MAPREDUCE-3255:
----------------------------------------

Since it would be a framework counter, though, we have no ability to ensure that different
containers have the same resource utilization. And comparing across applications (eg for organizational
billing), we need a standardized unit.

Using RAM-seconds now doesn't preclude us from adding more counters for other resource types
later. I imagine that, for any resource type that is requested/reserved by a job, we'd have
a counter for that resource.

This also allows us future improvements where the resource requests for a container may even
change throughout its life (eg for native apps where changing heap usage is easy to do)
                
> MR2 needs a counter analogous to "slot seconds"
> -----------------------------------------------
>
>                 Key: MAPREDUCE-3255
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3255
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>
> In older versions of MR, we have a SLOT_MILLIS counter which counts how many seconds
each task (and job) used of map/reduce slots. Since we no longer have the concept of a "slot"
in MR2, we should change this counter to be something like "RAM-megabyte-seconds" - ie each
task is charged for its resource requirement * the number of seconds it occupied those resources.
> Ideally these would be collected by the NodeManager rather than the children, so that
they could be used for billing purposes, but a first cut of the counters through the MR framework
itself would probably still be useful in more trusting environments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message