hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Boris Shkolnik (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1430) JobTracker should be able to renew delegation tokens for the jobs
Date Thu, 18 Feb 2010 02:55:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835099#action_12835099
] 

Boris Shkolnik commented on MAPREDUCE-1430:
-------------------------------------------

bq.1) For (3), it should be that, cancellation is done for tokens owned by jobs that have
mapreduce.job.cancel.delegation.tokens set to true, when the job is done. The idea is to immediately
cancel the tokens once the job is done unless the user sets mapreduce.job.cancel.delegation.tokens
to false (in which case it probably means that the user launched jobs from tasks of the job
in question, and these new jobs would be using the same tokens for talking to the hdfs). For
cancelling the token you should invoke the appropriate RPC. Thinking about it, the config
could be named to something like "mapreduce.job.complete.cancel.delegation.tokens"
Added cancellation for completed jobs (unless "mapreduce.job.complete.cancel.delegation.tokens"
is set to false (default is true)

bq. 2) Please make the DelegationTokenRenewal class package private.
It is used from JobTracker so it must remain public. Added  @InterfaceAudience.Private.

bq. 3) renewalTimer should be a daemon thread.
Done.

bq. 4) Methods like getDfsForToken could be declared as private.
Done.

bq. 5) The logic for setting the renewal to 60 minutes on an exception should be within the
renewDelegationToken method (newExpirationDate could be initialized to that).
Done.

bq. 6) You don't need synchronization on delegationTokens in the setTimerForTokenRenewal method.
Done.

bq. 7) The delegationTokens list could be a HashSet instead
Actually, if we want direct access it should be a MAP keyed by jobId. And that will point
to a list. So I changed it to Map<String, List<DTToRenew>>

I've also
8) modified Test to verify cancellation of the token.
9) moved registration for renewal to job initialization and cancelation to garbageCollect()
10) specified default for ""mapreduce.job.complete.cancel.delegation.tokens" to true in mapred-default.xml
11) change all Boolean renew/cancelToken to long

and more..

> JobTracker should be able to renew delegation tokens for the jobs
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1430
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1430
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: jobtracker
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: MAPREDUCE-1430-5.patch, MAPREDUCE-1430-6.patch, MAPREDUCE-1430-8.patch
>
>
> JobTracker should automatically renew delegation tokens for the jobs it is currently
running.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message