hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
Date Mon, 15 Oct 2012 16:38:03 GMT

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476251#comment-13476251

Alejandro Abdelnur commented on YARN-2:

*Num cores...*

I'd prefer to use a float instead of a int because it will avoid user confusion on what the
integer part of the number means, specially when dealing with clusters with large number of
cores in their nodes.

*NM memory and CPU configs*

Currently these values are coming from the config of the NM, we should be able to obtain those
values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this
is highly OS dependent we should have an interface that obtains this information. In addition
implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu
not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other
services outside of YARN containers. 

I did a quick search and I couldn't find a JIRA for this, opening YARN-160.
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.3-alpha
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch,
YARN-2-help.patch, YARN-2.patch, YARN-2.patch
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message