hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3255) MR2 needs a counter analogous to "slot seconds"
Date Tue, 25 Oct 2011 17:20:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135246#comment-13135246

Todd Lipcon commented on MAPREDUCE-3255:

Since it would be a framework counter, though, we have no ability to ensure that different
containers have the same resource utilization. And comparing across applications (eg for organizational
billing), we need a standardized unit.

Using RAM-seconds now doesn't preclude us from adding more counters for other resource types
later. I imagine that, for any resource type that is requested/reserved by a job, we'd have
a counter for that resource.

This also allows us future improvements where the resource requests for a container may even
change throughout its life (eg for native apps where changing heap usage is easy to do)
> MR2 needs a counter analogous to "slot seconds"
> -----------------------------------------------
>                 Key: MAPREDUCE-3255
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3255
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
> In older versions of MR, we have a SLOT_MILLIS counter which counts how many seconds
each task (and job) used of map/reduce slots. Since we no longer have the concept of a "slot"
in MR2, we should change this counter to be something like "RAM-megabyte-seconds" - ie each
task is charged for its resource requirement * the number of seconds it occupied those resources.
> Ideally these would be collected by the NodeManager rather than the children, so that
they could be used for billing purposes, but a first cut of the counters through the MR framework
itself would probably still be useful in more trusting environments.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message