hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2) Enhance CS to schedule accounting for both memory and cpu cores
Date Sun, 09 Sep 2012 02:02:10 GMT

    [ https://issues.apache.org/jira/browse/YARN-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451453#comment-13451453

Vinod Kumar Vavilapalli commented on YARN-2:

Alright, the patch needs some minor upmerge to the latest trunk. Other than that, a few comments:
lots of text, but mostly minor issues:

 - api.records.Resource should perhaps be not comparable anymore after these changes?

 - It seems unnatural for the comparator to store {{clusterResource}}, is it possible to just
pass in {{clusterResource}} where-ever needed? I see {{compare()}} -> {{getResourceAsValue()}}
and {{divide()}} -> {{getResourceAsValue()}} using this. For all the traces of divide(),
I verified that {{clusterResource}} can be passed all the way through. For {{compare()}} method,
see below.
 - Haven't read the paper yet, but in the current impl, we may end up comparing cpus to memory
when comparing two resources. Is that intended? If not, even compare() won't need to read
 - {{ResourceComparator}} is doing much more than comparisons - may be call it something like
 - {{divide()}} seems to be completely unrelated to {{divideAndCeil()}} looking at the implementation.
One of the names need to change, not sure which.
 - Not sure how to differentiate between {{ratio()}} and {{divide()}} either. The implementation
does differ a lot, I couldn't figure out which old references of the division operator were
replaced by {{divide()}} and which ones by {{ratio()}}. Names can be made more explicit, if
not that, at the least the documentation.

 -  {{multiplyAndNormalizeUp()}} and {{multiplyAndNormalizeDown()}}, {{roundUp()}}, {{roundDown()}},
ratio(), divide(), divideAndCeil() and equals() are unnecessary level of indirections to ResourceComparator
and can be removed? lessThan(), lessThanOrEqual(), greaterThan(), greaterThanOrEqual(), min(),
max() are useful though.
 - createResource(int memory) constructor is going to be problem when other resources/scheduling-algos
come in. We should rename it, but okay doing it later.

bq. Could we change the name of ResourceMemoryCpuComparator to something more like DefaultMultiResourceComparator?
I think ResourceMemoryCpuNetworkBandwithDiskStorageGPUComparator is a bit long, but it is
the direction we are headed in.
I too agree. Given this is also going to be public (albeit admin facing) configuration, I
am +1 for something like MultiResourceComparator. Thoughts?

Problems worth considering in follow-up JIRAs:
 - Num cores to be a float to allow a bit of over-subscription like it is possible today without
any concept of cores. Granted this can be simulated by artificially increasing the number
of cores in configuration, but we should think of a more appropriate way.
 - Some calculations like in LeafQueue have become very hard to read, we can rewrite them
to calculate one item per line :)
> Enhance CS to schedule accounting for both memory and cpu cores
> ---------------------------------------------------------------
>                 Key: YARN-2
>                 URL: https://issues.apache.org/jira/browse/YARN-2
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacityscheduler, scheduler
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 2.0.2-alpha
>         Attachments: MAPREDUCE-4327.patch, MAPREDUCE-4327.patch, MAPREDUCE-4327.patch,
MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch,
YARN-2-help.patch, YARN-2.patch, YARN-2.patch
> With YARN being a general purpose system, it would be useful for several applications
(MPI et al) to specify not just memory but also CPU (cores) for their resource requirements.
Thus, it would be useful to the CapacityScheduler to account for both.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message