hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Radim Kolar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4256) Improve resource scheduling
Date Sat, 20 Oct 2012 13:08:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480731#comment-13480731

Radim Kolar commented on MAPREDUCE-4256:

About optional resources like GPU for computation. Lets task to specify weight in application
resource requests.

for example request like "gpu:1:.8" means requesting 1 of gpu resources with weight 0.8 ->
give node 80% bonus if gpu resource is available

it could handle data locality as well, because for some compute tasks data locality its not
needed because you do not move much data over net. In that case request will be "data:1:0.0"
> Improve resource scheduling
> ---------------------------
>                 Key: MAPREDUCE-4256
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4256
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Radim Kolar
> Currently resource manager supports only Memory resource during container allocation.
> I propose following improvements:
> 1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin.
> 2. add support for custom resources. In node configuration will be something like:
> name=node.resource.GPU, value=1 (node has 1 GPU).
> If job will need to use GPU for computation, it will add "GPU=1" requirement to its job
config and Resource Manager will allocate container on node with GPU available.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message