hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
Date Mon, 24 Dec 2012 04:34:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539180#comment-13539180
] 

Arun C Murthy commented on YARN-2:
----------------------------------

Ok, I finally got around to finishing this up.

W.r.t cores, on some more thinking, I'm inclined to go along with the concept of integral
'virtual cores' instead of a float-precision 'cores' for the following reasons:
* It provides a level on indirection to deal with heterogenous cores viz. much more important
for CPUs (as opposed to memory, disk b/w etc.). I've also added a notion of physical-to-virtual
cores translation (yarn.nodemanager.vcores-pcores-ratio), per NodeManager, similar to physical-to-virtual
memory translation (yarn.nodemanager.vmem-pmem-ratio) that we already have in place.
* It ensures we do minimal floating-point operations in the inner-most loop which are very
expensive (for e.g. we dropped usage of Math.ceil in MAPREDUCE-1354 for JobTracker - Math.ceil
is a JNI call). This is something I've been very focussed on since the dawn, which explains
the integral implementations of divideAndCeil we've had since the beginning.

To make it clear, I've also renamed the apis to be (get,set)VirtualCores and marked them evolving
- in future we can add (get,set)Cores after we finalize how we specify not just #cores, but
also their capabilities (gigahertz?).

Thoughts?
                
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.3-alpha
>
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch,
YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message