hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Jie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15419) Should not obtain delegationTokens from all namenodes when using ViewFS
Date Fri, 27 Apr 2018 06:48:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456005#comment-16456005

Tao Jie commented on HADOOP-15419:

[~hexiaoqiao] thank you for your comment.
In fact, some request path may be found after task running (for one simple instance: hard
code path in MapReduce/Spark job), if we don't obtain delegation token for that NameNode,
the Job will be fail due to all tasks can not pass authentication.
Agree. For the cluster, it is not easy to know which namenodes a certain job would access.
I think the mechanism could be more flexible and obtaining tokens from all namenodes seems
to be too crude.
1, We can have a option maybe {{fs.viewfs.use.specific.filesystem}}, only when this option
is true, the following logic works.
2, When submit a mr/spark job, if the input/output path is a viewfs path, instead of obtaining
token from all namenode, we would visit and fetch token from only a SET of filesystem.
3, The raw filesystem of the input/output path should be in the SET
4, We may have a global option like {{fs.viewfs.global.filesystem}} which defines filesystems
that all jobs may visit(Eg. the filesystem of tmp dir, scratch dir), and it should be added
into the SET
5, Job-level option like {{fs.viewfs.additional.filesystem}} which defiles extra filesystem
that the certain job need.
Since obtaining delegation tokens happens on the client side, the effect of the modification
would be controllable.  
Any thought?

> Should not obtain delegationTokens from all namenodes when using ViewFS
> -----------------------------------------------------------------------
>                 Key: HADOOP-15419
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15419
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.0.2
>            Reporter: Tao Jie
>            Priority: Major
> Today when submit a job to a viewfs cluster, the client will try to obtain delegation
token from all namenodes under the viewfs while only one namespace is actually used in this
job. It would create many unnecessary rpc call to the whole cluster.
> In viewfs situation, we can just obtain delegation token from specific namenode rather
than all namenodes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message