hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2791) Add Disk as a resource for scheduling
Date Tue, 11 Nov 2014 20:21:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206993#comment-14206993
] 

Karthik Kambatla commented on YARN-2791:
----------------------------------------

Thanks [~sdaingade] for sharing the design doc. Well articulated. 

The designs on YARN-2139 and YARN-2791 are very similar, except for the disk resources are
called vdisks in YARN-2139 and spindles in YARN-2791. In addition to the items specified here,
YARN-2139 talks about isolation as well. Other than that, do you see any major items YARN-2791
covers that YARN-2139? The WebUI is good and very desirable, we should definitely include
it. Also, 

I suggest we make this (as is - or split into multiple JIRAs) a sub-task of YARN-2139. Discussing
the high-level details on one JIRA helps with aligning on one final design doc based on everyone's
suggestions. 



> Add Disk as a resource for scheduling
> -------------------------------------
>
>                 Key: YARN-2791
>                 URL: https://issues.apache.org/jira/browse/YARN-2791
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>    Affects Versions: 2.5.1
>            Reporter: Swapnil Daingade
>            Assignee: Yuliya Feldman
>         Attachments: DiskDriveAsResourceInYARN.pdf
>
>
> Currently, the number of disks present on a node is not considered a factor while scheduling
containers on that node. Having large amount of memory on a node can lead to high number of
containers being launched on that node, all of which compete for I/O bandwidth. This multiplexing
of I/O across containers can lead to slower overall progress and sub-optimal resource utilization
as containers starved for I/O bandwidth hold on to other resources like cpu and memory. This
problem can be solved by considering disk as a resource and including it in deciding how many
containers can be concurrently run on a node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message