Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Thu, 30 Aug 2012 07:49:09 +1100 (NCT)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <1280156497.13705.1346273350021.JavaMail.jiratomcat@arcas>
In-Reply-To: <218805250.12762.1346262369070.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (YARN-60) NMs rejects all container tokens after
 2nd secret key roll
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-60?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444396#comment-13444396 ] 

Daryn Sharp commented on YARN-60:
---------------------------------

RM appears to be throwing NPE in response to the NM heartbeat.  The NMs never get the new key, so when it's activated, the NMs reject all tokens.
                
> NMs rejects all container tokens after 2nd secret key roll
> ----------------------------------------------------------
>
>                 Key: YARN-60
>                 URL: https://issues.apache.org/jira/browse/YARN-60
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.2.0-alpha, 0.23.3
>            Reporter: Daryn Sharp
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Blocker
>
> The NM's token secret manager will reject all container tokens after the third secret key is activated which means the NM will not launch _any_ containers including AMs.  The whole yarn cluster becomes inoperable in 2d 15m (default 1d key roll + 15m activation delay).
> The NM tracks the previous 2 secrets, so either the RM is generating tokens with an old secret, or more likely the NMs are going out of sync when discarding the oldest secret.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira