hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Yan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2139) Add support for disk IO isolation/scheduling for containers
Date Wed, 02 Jul 2014 16:40:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050301#comment-14050301
] 

Wei Yan commented on YARN-2139:
-------------------------------

Thanks for the comments, [~stevel@apache.org].
For you mentioned HDFS read/write problem, we leave it solved by the network part, as we also
need handle the hdfs replicate traffic. I agree that we should avoid confuction with HDFS
''fs''.

The idea of vdisks follows the vcores, where each physical cpu core is measured as some number
of vcores. One concern about using real number is that users cannot specify their task requirements
easily. One way may solve that is to provide several levels (low, moderate, high, etc) instead
of real numbers. This is also similar to the discussions YARN-1024 on how to measure the cpu
capacity. We can define the how many IOPs/bandwidth map to 1 vdisks.

For the testability, currently I have: (1) For fairshare, start several tasks with same operations,
put them in a single node, and check their I/O performance whether follows fairsharing;  (2)
I/O performance isolation for a given task, in a fully loaded cluster, we replay the given
task several times, and verify when its I/O performance is stable. Here the task can do lots
of local disk read and directly write operation, and the most time is used to do the I/O.
Any good testing ideas?



> Add support for disk IO isolation/scheduling for containers
> -----------------------------------------------------------
>
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>            Assignee: Wei Yan
>         Attachments: Disk_IO_Scheduling_Design_1.pdf
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message