hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7872) labeled node cannot be used to satisfy locality specified request
Date Thu, 01 Feb 2018 13:46:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348591#comment-16348591
] 

genericqa commented on YARN-7872:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  8m 15s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
|| || || || {color:brown} branch-2.7.2 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  1m 51s{color} | {color:red}
root in branch-2.7.2 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 29s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 31s{color}
| {color:green} branch-2.7.2 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 17s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in branch-2.7.2 failed with JDK v9-internal. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m  9s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m  9s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m  9s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v9-internal. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v9-internal. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 25s{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 816 unchanged - 1 fixed = 818 total (was 817) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 11s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m  9s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v9-internal. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 10s{color} | {color:red}
hadoop-yarn-server-resourcemanager in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 22s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 17s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:date2018-02-01 |
| JIRA Issue | YARN-7872 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908800/YARN-7872-branch-2.7.2.001.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient
 findbugs  checkstyle  |
| uname | Linux 96c5df8dad1b 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7.2 / b165c4f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 9-internal |
| Multi-JDK versions |  /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_151 /usr/lib/jvm/java-9-openjdk-amd64:9-internal
|
| mvninstall | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-mvninstall-root.txt
|
| compile | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_151.txt
|
| compile | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_151.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| mvninstall | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_151.txt
|
| javac | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_151.txt
|
| compile | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| javac | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_151.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/19564/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk9-internal.txt
|
| JDK v9-internal  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19564/testReport/
|
| Max. process+thread count | 147 (vs. ulimit of 5000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19564/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> labeled node cannot be used to satisfy locality specified request
> -----------------------------------------------------------------
>
>                 Key: YARN-7872
>                 URL: https://issues.apache.org/jira/browse/YARN-7872
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, capacityscheduler, resourcemanager
>    Affects Versions: 2.7.2
>            Reporter: Yuqi Wang
>            Assignee: Yuqi Wang
>            Priority: Blocker
>             Fix For: 2.7.2
>
>         Attachments: YARN-7872-branch-2.7.2.001.patch
>
>
> *Issue summary:*
> labeled node (i.e. node with 'not empty' node label) cannot be used to satisfy locality
specified request (i.e. container request with 'not ANY' resource name and the relax locality
is false).
>  
> *For example:*
> The node with available resource:
> [Resource: [MemoryMB: [100] CpuNumber: [12]] {color:#14892c}NodeLabel: [persistent]{color}
{color:#f79232}HostName: \{SRG}{color} RackName: \{/default-rack}]
> The container request:
>  [Priority: [1] Resource: [MemoryMB: [1] CpuNumber: [1]] {color:#14892c}NodeLabel: [null]{color}
{color:#f79232}HostNames: \{SRG}{color} RackNames: {} {color:#59afe1}RelaxLocality: [false]{color}]
> Current RM capacity scheduler's behavior is that (at least for version 2.7 and 2.8),
the node cannot allocate container for the request, because the node label is not matched
when the leaf queue assign container.
>  
> *Possible solution:*
> However, node locality and node label should be two orthogonal dimensions to select
candidate nodes for container request. And the node label matching should only be executed
for container request with ANY resource name, since only this kind of container request is
allowed to have 'not empty' node label.
> So, for container request with 'not ANY' resource name (so, we clearly know it should not
have node label), we should use the requested resource name to match with the node instead
of using the requested node label to match with the node. And this resource name matching should
be safe, since the node whose node label is not accessible for the queue will not be sent to
the leaf queue.
>  
> *Discussion:*
> Attachment is the fix according to this principle, please help to review.
> Without it, we cannot use locality to request container within these labeled nodes.
> If the fix is acceptable, we should also recheck whether the same issue happens in trunk
and other hadoop versions.
> If not acceptable (i.e. the current behavior is by designed), so, how can we use locality
to request container within these labeled nodes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message