hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3856) YARN shoud allocate container that is closest to the data
Date Fri, 26 Jun 2015 08:19:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-3856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602550#comment-14602550
] 

Hadoop QA commented on YARN-3856:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  16m 54s | Pre-patch trunk has 1 extant Findbugs (version
3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 35s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 36s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 16s | The applied patch generated  19 new checkstyle
issues (total was 0, now 19). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that end in whitespace.
|
| {color:green}+1{color} | install |   1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with eclipse:eclipse.
|
| {color:red}-1{color} | findbugs |   2m 41s | The patch appears to introduce 1 new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m  3s | Tests passed in hadoop-mapreduce-client-app.
|
| {color:green}+1{color} | yarn tests |   1m 56s | Tests passed in hadoop-yarn-common. |
| | |  51m 33s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-common |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12742067/YARN-3856.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8ef07f7 |
| Pre-patch Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8356/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| checkstyle |  https://builds.apache.org/job/PreCommit-YARN-Build/8356/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8356/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
|
| hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/8356/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/8356/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8356/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8356/console |


This message was automatically generated.

> YARN shoud allocate container that is closest to the data
> ---------------------------------------------------------
>
>                 Key: YARN-3856
>                 URL: https://issues.apache.org/jira/browse/YARN-3856
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>         Environment: Hadoop cluster with multi-level network hierarchy
>            Reporter: jaehoon ko
>         Attachments: YARN-3856.001.patch
>
>
> Currently, given a Container request for a host, ResourceManager allocates a Container
with following priorities (RMContainerAllocator.java):
>  - Requested host
>  - a host in the same rack as the requested host
>  - any host
> This can lead to a sub-optimal allocation if Hadoop cluster is deployed on multi-level
networked hosts (which is typical). For example, let's suppose a network architecture with
one core switches, two aggregate switches, four ToR switches, and 8 hosts. Each switch has
two downlinks. Rack IDs of hosts are as follows:
> h1, h2: /c/a1/t1
> h3, h4: /c/a1/t2
> h5, h6: /c/a2/t3
> h7, h8: /c/a2/t4
> To allocate a container for data in h1, Hadoop first tries h1 itself, then h2, then any
of h3 ~ h8. Clearly, h3 or h4 are better than h5~h8 in terms of network distance and bandwidth.
However, current implementation choose one from h3~h8 with equal probabilities.
> This limitation is more obvious when considering hadoop clusters deployed on VM or containers.
In this case, only the VMs or containers running in the same physical host are considered
rack local, and actual rack-local hosts are chosen with same probabilities as far hosts.
> The root cause of this limitation is that RMContainerAllocator.java performs exact matching
on rack id to find a rack local host. Alternatively, we can perform longest-prefix matching
to find a closest host. Using the same network architecture as above, with longest-prefix
matching, hosts are selected with the following priorities:
>  h1
>  h2
>  h3 or h4
>  h5 or h6 or h7 or h8



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message