hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Radim Kolar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4256) Improve resource scheduling
Date Sun, 10 Jun 2012 18:50:44 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292554#comment-13292554
] 

Radim Kolar commented on MAPREDUCE-4256:
----------------------------------------

1 would leave optional resources and other stuff like tiered resources and data locality vs
optional resources for Resource scheduling for hadoop-3. For hadoop-2 lets start with something
simple.

1. handling different OS - one plugin for each operation system. If plugin runs on unsupported
OS, it will throw exception during @PostConstruct method.

2. Resource restrictions can be supported if resource plugin will have optional function implemented
- query how much resource of that type is currently in use by container.

3. Good point are resources which can change without any input from RM. Lets call them volatile
resources - if resources is marked as such, RM will need to do periodic queries.

4. network accessible resources will be handled as global resources
                
> Improve resource scheduling
> ---------------------------
>
>                 Key: MAPREDUCE-4256
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4256
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: resourcemanager
>            Reporter: Radim Kolar
>
> Currently resource manager supports only Memory resource during container allocation.
> I propose following improvements:
> 1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin.
> 2. add support for custom resources. In node configuration will be something like:
> name=node.resource.GPU, value=1 (node has 1 GPU).
> If job will need to use GPU for computation, it will add "GPU=1" requirement to its job
config and Resource Manager will allocate container on node with GPU available.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message