hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Radim Kolar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4256) Improve resource scheduling
Date Mon, 21 May 2012 09:25:41 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280049#comment-13280049

Radim Kolar commented on MAPREDUCE-4256:

Two kind of resources:

 1. cluster shared resources - every node in cluster can use them. For example WAN bandwidth.

 2. node local resources - for example GPU card.

these resources will be configured using standard hadoop configuration framework.

mapreduce.resources.cluster.available=resourcename:float,... (this option will be used by
mapreduce.resources.node.available=resourcename:float (for use by nodemanager). Node manager
will report available resources to resourcemanager

job config:
mapreduce.resources.mapper= list of resources needed by one mapper
mapreduce.resources.reducer= list of resources needed by one reducer

if same resource is configured per node and per cluster then per node version is preferred.
> Improve resource scheduling
> ---------------------------
>                 Key: MAPREDUCE-4256
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4256
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Radim Kolar
>             Fix For: 2.0.0, 3.0.0
> Currently resource manager supports only Memory resource during container allocation.
> I propose following improvements:
> 1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin.
> 2. add support for custom resources. In node configuration will be something like:
> name=node.resource.GPU, value=1 (node has 1 GPU).
> If job will need to use GPU for computation, it will add "GPU=1" requirement to its job
config and Resource Manager will allocate container on node with GPU available.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message