hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12357) Let NameNode to bypass external attribute provider for special user
Date Thu, 31 Aug 2017 01:15:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148139#comment-16148139
] 

Yongjun Zhang edited comment on HDFS-12357 at 8/31/17 1:14 AM:
---------------------------------------------------------------

Thanks you all for the review and comments!

[~atm]: good point, will remove the static. Thanks.

[~daryn], thanks for your comments, some thoughts:
1. Based on user/path to decide what attributes to reveal is indeed more refined. However,
it adds complexity. And every provider has to provide an implementation. Wonder if you can
provide an example we want to decide things based on user/path combination?
2. Currently I use NameNode.getRemoteUser() to tell which user it is. If we put this bypass
logic into Provider, the provider need to know what the current user is. we either have to
change the API of provider, or add some new methods in parallel, to pass the user information.


[~manojg], talking about SnapshotDiff to bypass provider, the caller need to tell the provider
to do that, thus new API is needed. Right? thanks.

Look forward to your further thoughts and comments!

Thanks a lot.




was (Author: yzhangal):
Thanks you all for the review and comments!

[~atm]: good point, will remove the static. Thanks.

[~daryn], thanks for your comments, some thoughts:
1. Based on user/path to decide what attributes to reveal is indeed more refined. However,
it adds complexity. And every provider has to provide an implementation. Wonder if you can
provide an example we want to decide things based on user/path?
2. Currently I use NameNode.getRemoteUser() to tell which user it is. If we put this bypass
logic into Provider, the provider need to know what the current user is. we either have to
change the API of provider, or add some new methods in parallel.

[~manojg], talking about SnapshotDiff to bypass provider, the caller need to tell the provider
to do that, thus new API is needed. Right? thanks.

Look forward to your further thoughts and comments!

Thanks a lot.



> Let NameNode to bypass external attribute provider for special user
> -------------------------------------------------------------------
>
>                 Key: HDFS-12357
>                 URL: https://issues.apache.org/jira/browse/HDFS-12357
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-12357.001.patch
>
>
> This is a third proposal to solve the problem described in HDFS-12202.
> The problem is, when we do distcp from one cluster to another (or within the same cluster),
in addition to copying file data, we copy the metadata from source to target. If external
attribute provider is enabled, the metadata may be read from the provider, thus provider data
read from source may be saved to target HDFS. 
> We want to avoid saving metadata from external provider to HDFS, so we want to bypass
external provider when doing the distcp (or hadoop fs -cp) operation.
> Two alternative approaches were proposed earlier, one in HDFS-12202, the other in HDFS-12294.
The proposal here is the third one.
> The idea is, we introduce a new config, that specifies a special user (or a list of users),
and let NN bypass external provider when the current user is a special user.
> If we run applications as the special user that need data from external attribute provider,
then it won't work. So the constraint on this approach is, the special users here should not
run applications that need data from external provider.
> Thanks [~asuresh] for proposing this idea and [~chris.douglas], [~daryn], [~manojg] for
the discussions in the other jiras. 
> I'm creating this one to discuss further.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message