hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
Date Sat, 29 Dec 2012 03:48:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13540746#comment-13540746
] 

Arun C Murthy commented on YARN-2:
----------------------------------

[~sandyr] See the discussion between Bikas & I w.r.t definition of a virtual-core. That
should take care of your concerns.

Here is AWS's take on an EC2 Compute Unit viz. similar to the 'virtual core': http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it

However, I don't think we are in a position, yet, to define a virtual-core with certainty
(should we choose a 2010 Xeon or a bleeding edge 2012 X5000 series?) - hence the leeway for
admins until we get more experience with this.

IAC, as [~revans2] pointed out, we can debate endlessly or run with something reasonable for
now - the apis are marked @Evolving, and we'll likely change once we learn more. Essentially,
cpu-scheduling is an experimental feature for now.
                
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.3-alpha
>
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch,
YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch,
YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message