spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Long running Spark job on YARN throws "No AMRMToken"
Date Sat, 13 Feb 2016 15:13:10 GMT

On 11 Feb 2016, at 15:24, Prabhu Joseph <prabhujose.gates@gmail.com<mailto:prabhujose.gates@gmail.com>>
wrote:

Steve,


      When ResourceManager is submitted with an application, AMLauncher creates the token
YARN_AM_RM_TOKEN (token used between RM and AM). When ApplicationMaster
is launched, it tries to contact RM for registering request, allocate request to receive containers,
finish request. In all the requests,

yes, see

https://github.com/steveloughran/hadoop-trunk/blob/HADOOP-12649-security/YARN-4653-yarn/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md


ResourceManager does the
authorizeRequest, where it checks if the Current User has the token YARN_AM_RM_TOKEN, if not
throws the "No AMRMToken".

yes; prior to YARN-3103 it did the login user


       ResourceManager for every yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-sec
rolls the master key, before rolling it, it has a period
of 1.5 *  yarn.am.liveness-monitor.expiry-interval-ms during which if AM contacts RM with
allocate request, RM checks if the AM has the YARN_AM_RM_TOKEN
prepared using the previous master key, if so, it updates the AM user with YARN_AM_RM_TOKEN
prepared using new master key.

     If AM contacts with an YARN_AM_RM_TOKEN which is neither constructed using current master
key nor previous master key, then "Invalid AMRMToken" message is thrown. This
error is the one will happen if AM has not been updated with new RM master key. [YARN-3103
and YARN-2212 ]

Need your help to find scenario where "No AMRMToken" will happen, an user added with a token
but later that token is missing. Is token removed since expired?


...or there's some confusion about the current user

I've got a java class to help with credential creation and diagnostics, not yet ported to
hadoop core, which can do some listing & dumping of credentials

https://github.com/apache/incubator-slider/blob/develop/slider-core/src/main/java/org/apache/slider/core/launch/CredentialUtils.java

you may be able to copy that code and use it to print out what tokens the current user has;
otherwise I don't know. I've never personally hit the message

Mime
View raw message