hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
Date Wed, 26 Dec 2012 02:26:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539455#comment-13539455
] 

Bikas Saha commented on YARN-2:
-------------------------------

Could you please elaborate on your proposal of virtual cores a bit more. Specifically, around
your ideas for heterogeneous cores and over-subscription. That may help clarify some of the
questions raised in other comments.

Though I dont understand the suggestions fully, I would be wary of implicitly linking RM logic
with cgroups. Other than the unwritten dependency it also might make life harder for the ongoing
Windows port in YARN-191.

Also, other than the case for ~0 CPU tasks, what are the other scenarios for floating cores?
IMO we could just specify 0 cores for such tasks. Its safe because we cannot run an infinite
number of them because of other resource constraints like memory. I am not quite sure how/when
a non-integral CPU requirement would be needed.

                
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.3-alpha
>
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch,
YARN-2-help.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message