hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Tianyi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2139) [Umbrella] Support for Disk as a Resource in YARN
Date Sat, 30 Jan 2016 09:48:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124824#comment-15124824
] 

He Tianyi commented on YARN-2139:
---------------------------------

Recently introduced SSD in my cluster for MapReduce shuffle. 
Then there is one issue, if map output gets too large, it cannot be placed on SSD. We have
to implement a custom strategy (called SSDFirst) to make best effort to use SSD, but fallbacks
to HDD when available space of SSD gets tight. 
This worked in most cases, but it is only a local optimum. To achieve global optimum, scheduler
must be aware and management these resources.

> [Umbrella] Support for Disk as a Resource in YARN 
> --------------------------------------------------
>
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>         Attachments: Disk_IO_Isolation_Scheduling_3.pdf, Disk_IO_Scheduling_Design_1.pdf,
Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on nodes, (2)
isolation at runtime, (3) spindle locality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message