hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-503) DelegationTokens will be renewed forever if multiple jobs share tokens and the first one sets JOB_CANCEL_DELEGATION_TOKEN to false
Date Tue, 26 Mar 2013 00:09:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613321#comment-13613321
] 

Alejandro Abdelnur commented on YARN-503:
-----------------------------------------

bq. Oozie should be unaffected because as long as the launcher job or any child job is running
the tokens will remain valid. Alejandro Abdelnur, can you confirm?

In the case of Oozie MR actions, the launcher job ends as soon as the MR job is started, then
Oozie tracks the real MR job.

For all other actions (pig, hive, sqoop, distcp, java, shell) the launcher job blocks (because
of the corresponding Main class) until the child/ren job/s end.

Will the case of MR actions be an issue? that the launcher goes away?

                
> DelegationTokens will be renewed forever if multiple jobs share tokens and the first
one sets JOB_CANCEL_DELEGATION_TOKEN to false
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-503
>                 URL: https://issues.apache.org/jira/browse/YARN-503
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 0.23.3, 3.0.0, 2.0.0-alpha
>            Reporter: Siddharth Seth
>            Assignee: Daryn Sharp
>         Attachments: YARN-503.patch
>
>
> The first Job/App to register a token is the one which DelegationTokenRenewer associates
with a a specific Token. An attempt to remove/cancel these shared tokens by subsequent jobs
doesn't work - since the JobId will not match.
> As a result, Even if subsequent jobs have MRJobConfig.JOB_CANCEL_DELEGATION_TOKEN set
to true - tokens will not be cancelled when those jobs complete.
> Tokens will eventually be removed from the RM / JT when the service that issued them
considers them to have expired or via an explicit cancelDelegationTokens call (not implemented
yet in 23).
> A side affect of this is that the same delegation token will end up being renewed multiple
times (a separate TimerTask for each job which uses the token).
> DelegationTokenRenewer could maintain a reference count/list of jobIds for shared tokens.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message