hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6093) Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken renewal during a RM failover
Date Wed, 15 Feb 2017 01:02:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867030#comment-15867030
] 

Hadoop QA commented on YARN-6093:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 27s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 26s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m  9s{color}
| {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 28s{color} |
{color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 55s{color}
| {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 57s{color} |
{color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 44s{color}
| {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 33s{color} |
{color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 44s{color} |
{color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 10s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 39s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  3s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m  3s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 43s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 53s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 36s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 35s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 41s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 17s{color} | {color:red}
hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m 26s{color} | {color:green}
hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 33s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 32s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.federation.policies.router.TestWeightedRandomRouterPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6093 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12852692/YARN-6093-YARN-2915.v4.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux d555cfc3f872 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-2915 / 08dc095 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-YARN-Build/14949/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/14949/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/14949/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Invalid AMRM token exception when using FederationRMFailoverProxyProvider at AMRMtoken
renewal during a RM failover
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6093
>                 URL: https://issues.apache.org/jira/browse/YARN-6093
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: amrmproxy, federation
>    Affects Versions: YARN-2915
>            Reporter: Botong Huang
>            Assignee: Botong Huang
>            Priority: Minor
>         Attachments: YARN-6093-08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch, YARN-6093-git08dc09581230ba595ce48fe7d3bc4eb2b6f98091.v4.patch,
YARN-6093.v1.patch, YARN-6093-YARN-2915.v1.patch, YARN-6093-YARN-2915.v2.patch, YARN-6093-YARN-2915.v3.patch,
YARN-6093-YARN-2915.v4.patch
>
>
> AMRMProxy uses expired AMRMToken to talk to RM, leading to the "Invalid AMRMToken" exception.
The bug is triggered when both conditions are met: 
> 1. RM rolls master key and renews AMRMToken for a running AM.
> 2. Existing RPC connection between AMRMProxy and RM drops and attempt to reconnect via
failover in FederationRMFailoverProxyProvider. 
> Here's what happened: 
> In DefaultRequestInterceptor.init(), we create a proxy ugi, load it with the initial
AMRMToken issued by RM, and used it for initiating rmClient. Then we arrive at FederationRMFailoverProxyProvider.init(),
a full copy of ugi tokens are saved locally, create an actual RM proxy and setup the RPC connection.

> Later when RM rolls master key and issues a new AMRMToken, DefaultRequestInterceptor.updateAMRMToken()
updates it into the proxy ugi. 
> However the new token is never used, until the existing RPC connection between AMRMProxy
and RM drops for other reasons (say master RM crashes). 
> When we try to reconnect, since the service name of the new AMRMToken is not yet set
correctly in DefaultRequestInterceptor.updateAMRMToken(), RPC found no valid AMRMToken when
trying to setup a new connection. We first hit a "Client cannot authenticate via:[TOKEN]"
exception. This is expected. 
> Next, FederationRMFailoverProxyProvider fails over, we reset the service token via ClientRMProxy.getRMAddress()
and reconnect. Supposedly this would have worked. 
> However since DefaultRequestInterceptor does not use the proxy user for later calls to
rmClient, when performing failover in FederationRMFailoverProxyProvider, we are not in the
proxy user. Currently the code solve the problem by reloading the current ugi with all tokens
saved locally in originalTokens in method addOriginalTokens(). The problem is that the original
AMRMToken loaded is no longer accepted by RM, and thus we keep hitting the "Invalid AMRMToken"
exception until AM fails. 
> The correct way is that rather than saving the original tokens in the proxy ugi, we save
the original ugi itself. Every time we perform failover and create the new RM proxy, we use
the original ugi, which is always loaded with the up-to-date AMRMToken. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message