hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2420) JobTracker should be able to renew delegation token over HTTP
Date Tue, 05 Apr 2011 18:19:05 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016060#comment-13016060

Devaraj Das commented on MAPREDUCE-2420:

This patch handles the case of HDFS token renewals, when the JobTracker's HDFS is running
a different version of HDFS than the one the job is trying to use. An example of such a job
is distcp (where it uses hftp to talk to a different source cluster to pull data to the cluster
where distcp is running).

When the job is submitted, the client requests a delegation token over hftp and stuffs it
in the job. Today, the NameNode doesn't distinguish between hftp and hdfs accesses, and issues
HDFS tokens for both (and the token-kind field in the token has the value as 'HDFS'). Ideally,
that should be fixed to have the token-kind as HFTP for hftp accesses. We should have the
JobTracker handle all sorts of token renewals, and have a way in which it can look at a token
and decide which protocol to use to talk to the server in question. This includes HDFS, HFTP,
and also HIVE (where the protocol is thrift). 

I think this patch is okay for the short term - the JobTracker falls back to hftp if it couldn't
renew a token over hdfs. In the patch, there are a bunch of white space changes that aren't
required. The string comparisons for exception messages and then instantiating a concrete
exception could probably be replaced with a forname() on the string exception.

When we fix this issue in trunk, please make it more generic on lines similar to above.

> JobTracker should be able to renew delegation token over HTTP
> -------------------------------------------------------------
>                 Key: MAPREDUCE-2420
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2420
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.4
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: MR-2420.patch
> in case JobTracker has to talk to a NameNode running a different version (RPC version
mismatch), Jobtracker should be able to fall back to HTTP renewal.
> Example of the case - running distcp between different versions using hfpt.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message