hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9525) hadoop utilities need to support provided delegation tokens
Date Tue, 19 Jan 2016 18:34:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107148#comment-15107148

Daryn Sharp commented on HDFS-9525:

I think I understand better what you are trying to do, and I think you might be able to accomplish
your goals without much if any code change.   I think the main source of frustration is trying
to access a secure cluster with security disabled?

If you are trying to access any secure cluster: enable security in the config.  If you will
also access an insecure cluster: also set ipc.client.fallback-to-simple-auth-allowed=true.
 Now you should be able to access any mixture of (in)secure clusters using hdfs or webhdfs.

There's also an existing config "mapreduce.job.credentials.binary" that can be used to read
in a token cache.

Aside: If using webhdfs for both source and target, I'd advise against it.  Webhdfs generates
a much higher load on a cluster and is much less fault-tolerant than normal hdfs.  Our rule
of thumb is always pull data (run distcp on the target), read source with webhdfs (but only
when  RPC is acl-ed off), always write to local target with hdfs.

If a code change is necessary, UGI should use {{Configuration#getTrimmedStrings}} and unconditionally
call {{Credentials.readTokenStorageFile}} instead of allowing the user to specify an invalid
setting.  Only webhdfs related change is {{WebHdfsFileSystem.canRefreshDelegationToken}} should
default to true.

> hadoop utilities need to support provided delegation tokens
> -----------------------------------------------------------
>                 Key: HDFS-9525
>                 URL: https://issues.apache.org/jira/browse/HDFS-9525
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: HeeSoo Kim
>            Priority: Blocker
>             Fix For: 3.0.0
>         Attachments: HDFS-7984.001.patch, HDFS-7984.002.patch, HDFS-7984.003.patch, HDFS-7984.004.patch,
HDFS-7984.005.patch, HDFS-7984.006.patch, HDFS-7984.007.patch, HDFS-7984.patch, HDFS-9525.008.patch,
> When using the webhdfs:// filesystem (especially from distcp), we need the ability to
inject a delegation token rather than webhdfs initialize its own.  This would allow for cross-authentication-zone
file system accesses.

This message was sent by Atlassian JIRA

View raw message