Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Sun, 9 Sep 2012 13:02:10 +1100 (NCT)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <1481719901.54837.1347156130505.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for
 both memory and cpu cores
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451453#comment-13451453 ] 

Vinod Kumar Vavilapalli commented on YARN-2:
--------------------------------------------

Alright, the patch needs some minor upmerge to the latest trunk. Other than that, a few comments: lots of text, but mostly minor issues:

 - api.records.Resource should perhaps be not comparable anymore after these changes?

ResourceComparator:
 - It seems unnatural for the comparator to store {{clusterResource}}, is it possible to just pass in {{clusterResource}} where-ever needed? I see {{compare()}} -> {{getResourceAsValue()}} and {{divide()}} -> {{getResourceAsValue()}} using this. For all the traces of divide(), I verified that {{clusterResource}} can be passed all the way through. For {{compare()}} method, see below.
 - Haven't read the paper yet, but in the current impl, we may end up comparing cpus to memory when comparing two resources. Is that intended? If not, even compare() won't need to read {{clusterResource}}.
 - {{ResourceComparator}} is doing much more than comparisons - may be call it something like {{ResourceSchedulingCalculator}}?
 - {{divide()}} seems to be completely unrelated to {{divideAndCeil()}} looking at the implementation. One of the names need to change, not sure which.
 - Not sure how to differentiate between {{ratio()}} and {{divide()}} either. The implementation does differ a lot, I couldn't figure out which old references of the division operator were replaced by {{divide()}} and which ones by {{ratio()}}. Names can be made more explicit, if not that, at the least the documentation.

Resources.java
 -  {{multiplyAndNormalizeUp()}} and {{multiplyAndNormalizeDown()}}, {{roundUp()}}, {{roundDown()}}, ratio(), divide(), divideAndCeil() and equals() are unnecessary level of indirections to ResourceComparator and can be removed? lessThan(), lessThanOrEqual(), greaterThan(), greaterThanOrEqual(), min(), max() are useful though.
 - createResource(int memory) constructor is going to be problem when other resources/scheduling-algos come in. We should rename it, but okay doing it later.


bq. Could we change the name of ResourceMemoryCpuComparator to something more like DefaultMultiResourceComparator? I think ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it is the direction we are headed in.
I too agree. Given this is also going to be public (albeit admin facing) configuration, I am +1 for something like MultiResourceComparator. Thoughts?

Problems worth considering in follow-up JIRAs:
 - Num cores to be a float to allow a bit of over-subscription like it is possible today without any concept of cores. Granted this can be simulated by artificially increasing the number of cores in configuration, but we should think of a more appropriate way.
 - Some calculations like in LeafQueue have become very hard to read, we can rewrite them to calculate one item per line :)
                
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.2-alpha
>
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, YARN-2-help.patch, YARN-2.patch, YARN-2.patch
>
>
> With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira