hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5683) Support specifying storage type for per-application local dirs
Date Fri, 20 Jan 2017 05:10:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831196#comment-15831196
] 

Hadoop QA commented on YARN-5683:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 13s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 20s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m  2s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m  7s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 53s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m  1s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 50s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 32s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  2s{color} |
{color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 20s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 13s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 58s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 58s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  1m 57s{color}
| {color:orange} root: The patch generated 7 new + 939 unchanged - 9 fixed = 946 total (was
948) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 17s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  2m 11s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 19s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 30s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 49s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 36s{color} | {color:green}
hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 34s{color} | {color:green}
hadoop-yarn-server-nodemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 11s{color} | {color:green}
hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 59s{color} | {color:green}
hadoop-mapreduce-client-app in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m 26s{color} | {color:green}
hadoop-mapreduce-client-jobclient in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 49s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}249m  2s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5683 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832871/YARN-5683-3.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux ca5f9793e4cb 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5d8b80e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/14714/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/14714/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/14714/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: . |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/14714/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Support specifying storage type for per-application local dirs
> --------------------------------------------------------------
>
>                 Key: YARN-5683
>                 URL: https://issues.apache.org/jira/browse/YARN-5683
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>              Labels: oct16-hard
>         Attachments: flow_diagram_for_MapReduce-2.png, flow_diagram_for_MapReduce.png,
YARN-5683-1.patch, YARN-5683-2.patch, YARN-5683-3.patch
>
>
> h3.  Introduction
> * Some applications of various frameworks (Flink, Spark and MapReduce etc) using local
storage (checkpoint, shuffle etc) might require high IO performance. It's useful to allocate
local directories to high performance storage media for these applications on heterogeneous
clusters.
> * YARN does not distinguish different storage types and hence applications cannot selectively
use storage media with different performance characteristics. Adding awareness of storage
media can allow YARN to make better decisions about the placement of local directories.
> h3.  Approach
> * NodeManager will distinguish storage types for local directories.
> ** yarn.nodemanager.local-dirs and yarn.nodemanager.log-dirs configuration should allow
the cluster administrator to optionally specify the storage type for each local directories.
Example: [SSD]/disk1/nm-local-dir,/disk2/nm-local-dir,/disk3/nm-local-dir (equals to [SSD]/disk1/nm-local-dir,[DISK]/disk2/nm-local-dir,[DISK]/disk3/nm-local-dir)
> ** StorageType defines DISK/SSD storage types and takes DISK as the default storage type.

> ** StorageLocation separates storage type and directory path, used by LocalDirAllocator
to aware the types of local dirs, the default storage type is DISK.
> ** getLocalPathForWrite method of LocalDirAllcator will prefer to choose the local directory
of the specified storage type, and will fallback to not care storage type if the requirement
can not be satisfied.
> ** Support for container related local/log directories by ContainerLaunch. All application
frameworks can set the environment variables (LOCAL_STORAGE_TYPE and LOG_STORAGE_TYPE) to
specified the desired storage type of local/log directories, and choose to not launch container
if fallback through these environment variables (ENSURE_LOCAL_STORAGE_TYPE and ENSURE_LOG_STORAGE_TYPE).
> * Allow specified storage type for various frameworks (Take MapReduce as an example)
> ** Add new configurations should allow application administrator to optionally specify
the storage type of local/log directories and fallback strategy (MapReduce configurations:
mapreduce.job.local-storage-type, mapreduce.job.log-storage-type, mapreduce.job.ensure-local-storage-type
and mapreduce.job.ensure-log-storage-type).
> ** Support for container work directories. Set the environment variables includes LOCAL_STORAGE_TYPE
and LOG_STORAGE_TYPE according to configurations above for ContainerLaunchContext and ApplicationSubmissionContext.
(MapReduce should update YARNRunner and TaskAttemptImpl)
> ** Add storage type prefix for request path to support for other local directories of
frameworks (such as shuffle directories for MapReduce). (MapReduce should update YarnOutputFiles,
MROutputFiles and YarnChild to support for output/work directories)
> ** Flow diagram for MapReduce framework
> !flow_diagram_for_MapReduce-2.png!
> h3.  Further Discussion
> * The requirement of storage type for local/log directories may not be satisfied on heterogeneous
clusters. To achieve global optimum, scheduler should aware and manage disk resources. [YARN-2139|https://issues.apache.org/jira/browse/YARN-2139]
is close to that but seems not support multiple storage types, maybe we should do even more
to aware the storage type of disk resource?
> * Node labels or node constraints ([YARN-3409|https://issues.apache.org/jira/browse/YARN-3409])
can also make a higher chance to satisfy the requirement of specified storage type.
> * Fallback strategy still needs to be concerned. Certain applications might not work
well when the requirement of storage type is not satisfied. When none of desired storage type
disk are available, should container launching be failed? let AM handle? We have implemented
a fallback strategy that fail to launch container when none of desired storage type disk are
available. Is there some better methods? 
> This feature has been used for half a year to meet the needs of some applications on
Alibaba search clusters.
> Please feel free to give your suggestions and opinions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message