Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 26 Mar 2013 20:27:16 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12545427.1331080235322.58956.1364329636480@arcas>
In-Reply-To: <JIRA.12545427.1331080235322@arcas>
References: <JIRA.12545427.1331080235322@arcas>
Subject: [jira] [Commented] (YARN-503) DelegationTokens will be renewed
 forever if multiple jobs share tokens and the first one sets
 JOB_CANCEL_DELEGATION_TOKEN to false
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614502#comment-13614502 ] 

Daryn Sharp commented on YARN-503:
----------------------------------

bq. Will the case of MR actions be an issue? that the launcher goes away?

No, the central focus of this patch is to keep tokens alive as long as _at least one job_ is using the tokens.  Upon job submission, the new app is immediately linked against the tokens.  So for an oozie action, it's ok for the launcher to exit after submitting an action.  The tokens will stay alive until the action, and any sub-jobs it may have launched, have completed.  After no app is running with the tokens, and the keepalive expires, the tokens are cancelled.

Note that by default I maintained 100% backwards compat in that tokens for oozie jobs setting the " mapreduce.job.complete.cancel.delegation.tokens=false" will never be cancelled.  The RM will stop renewing them and won't issue duplicate renews.  Until we deprecate/remove the setting, we may internally try make the conf setting a final to see what happens.

Will address findbugs after some webhdfs firefighting.
                
> DelegationTokens will be renewed forever if multiple jobs share tokens and the first one sets JOB_CANCEL_DELEGATION_TOKEN to false
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-503
>                 URL: https://issues.apache.org/jira/browse/YARN-503
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 0.23.3, 3.0.0, 2.0.0-alpha
>            Reporter: Siddharth Seth
>            Assignee: Daryn Sharp
>         Attachments: YARN-503.patch
>
>
> The first Job/App to register a token is the one which DelegationTokenRenewer associates with a a specific Token. An attempt to remove/cancel these shared tokens by subsequent jobs doesn't work - since the JobId will not match.
> As a result, Even if subsequent jobs have MRJobConfig.JOB_CANCEL_DELEGATION_TOKEN set to true - tokens will not be cancelled when those jobs complete.
> Tokens will eventually be removed from the RM / JT when the service that issued them considers them to have expired or via an explicit cancelDelegationTokens call (not implemented yet in 23).
> A side affect of this is that the same delegation token will end up being renewed multiple times (a separate TimerTask for each job which uses the token).
> DelegationTokenRenewer could maintain a reference count/list of jobIds for shared tokens.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira