Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Sat, 30 Jan 2016 09:48:40 +0000 (UTC)
From: "He Tianyi (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12720246.1402358576000.256718.1454147320969@Atlassian.JIRA>
In-Reply-To: <JIRA.12720246.1402358576000@Atlassian.JIRA>
References: <JIRA.12720246.1402358576000@Atlassian.JIRA>
 <JIRA.12720246.1402358576048@arcas>
Subject: [jira] [Commented] (YARN-2139) [Umbrella] Support for Disk as a
 Resource in YARN
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124824#comment-15124824 ] 

He Tianyi commented on YARN-2139:
---------------------------------

Recently introduced SSD in my cluster for MapReduce shuffle. 
Then there is one issue, if map output gets too large, it cannot be placed on SSD. We have to implement a custom strategy (called SSDFirst) to make best effort to use SSD, but fallbacks to HDD when available space of SSD gets tight. 
This worked in most cases, but it is only a local optimum. To achieve global optimum, scheduler must be aware and management these resources.

> [Umbrella] Support for Disk as a Resource in YARN 
> --------------------------------------------------
>
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>         Attachments: Disk_IO_Isolation_Scheduling_3.pdf, Disk_IO_Scheduling_Design_1.pdf, Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on nodes, (2) isolation at runtime, (3) spindle locality. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)