hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2618) Avoid over-allocation of disk resources
Date Tue, 09 Dec 2014 06:12:12 GMT

    [ https://issues.apache.org/jira/browse/YARN-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14239031#comment-14239031

Karthik Kambatla commented on YARN-2618:

Thanks Wei. Comments:
# I think we should avoid adding new methods to BuilderUtils. Resource.newInstance does the
necessary job. But for the number of places BuilderUtils is used, we want to get rid of it.

# Add vdisks to ResourceType. 
# DominantResourceCalculator#getResourceAsValue: We should probably leave the method's name
as is, and change the boolean argument to either an int or the ResourceType itself. 
# In DRC#computeAvailableContainers, we should add check-for-zero for memory and cpu as well.
Similarly, DRC#ratio should have zero-checks for memory and cpu as well.

bq. Capacity Scheduler already support multi-dimension resource by DominateResourceCalculator
and it should work when DRC updated to support disk.
I haven't looked at the CS code much, but it appears the scheduling decisions use DRC#divide
which only looks at memory. [~leftnoteasy] - any thoughts here? 

> Avoid over-allocation of disk resources
> ---------------------------------------
>                 Key: YARN-2618
>                 URL: https://issues.apache.org/jira/browse/YARN-2618
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Wei Yan
>            Assignee: Wei Yan
>         Attachments: YARN-2618-1.patch, YARN-2618-2.patch
> Subtask of YARN-2139. 
> This should include
> - Add API support for introducing disk I/O as the 3rd type resource.
> - NM should report this information to the RM
> - RM should consider this to avoid over-allocation

This message was sent by Atlassian JIRA

View raw message