hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Marquardt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15954) ABFS: Enable owner and group conversion for MSI and login user using OAuth
Date Fri, 07 Dec 2018 23:15:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713397#comment-16713397

Thomas Marquardt commented on HADOOP-15954:

I have a few ideas about how we should design this feature:

1) There should be a public interface named CustomIdentityTransformer in the org.apache.hadoop.fs.azurebfs.extensions

    a) This class is already implemented in patch 004, but the name should be changed from
CustomIdentityTransformerAdaptee to CustomIdentityTransformer.

    b) transformUserName and transformGroupName should be combined into a single function
named transformOwner if possible. Note that these methods are invoked to transform the input
of a FileSystem.setOwner call.

    c) All implementations of identity transformer will implement this interface, including the
default implementation.

2) The AbfsConfiguration.getIdentityTransformer method should do the following:

    a) If "*fs.azure.identity.transformer.type*" is not set, return immediately without
doing any additional work.

    b) If "*fs.azure.identity.transformer.type*" is set:

        i) Load the class.

        ii) Create a new instance.

        iii) Call initialize and pass in the configuration and account name.

        iv) Return the new initialized instance of CustomIdentityTransformer.

3) Remove the configuration "*fs.azure.account.identity.type*".  It is superfluous.

4) Remove the enum TransformerType.  It is superfluous since the "*fs.azure.identity.transformer.type*"
will define the type of identity transformer used.

5) Remove the class org.apache.hadoop.fs.azurebfs.oauth.CustomIdentityTransformerAdapter.

    a) The Adapter Design model is not appropriate here.  The Adapter Design model is used
when you have a public interface, but there are external component that implement a different
interface.  In our case there is no need to have an adapter, because the external component
can implement our new public interface directly.

6) Provide a default implementation named DefaultIdentityTransformer in the org.apache.hadoop.fs.azurebfs.oauth2

    a) This class is partly implemented, but:

        i) The name should be changed from StandardSPIdentityTransformer to DefaultIdentityTransformer.

        ii) DefaultIdentityTransformer should implement CustomIdentityTransformer. The
IdentityTransformer code should be moved into DefaultIdentityTransformer.


7) Configuration used by DefaultIdentityTransformer should be prefixed with “fs.azure.default.identity.transformer”:

    a) “*fs.azure.default.identity.transformer.enable.short.name*”  – user principal
names (UPNs) have the format “\{alias}@\{domain}”.  If true, only \{alias} is included
when a UPN would otherwise appear in the output of APIs like getFileStatus, getOwner, getAclStatus,

    b) “*fs.azure.default.identity.transformer.domain.name*” – If the domain name
is specified and “fs.azure.default.identity.transformer.enable.short.name” is true, then
the \{alias} part of a UPN can be specified as input to APIs like setOwner and setAcl and
it will be transformed to a UPN by appending @ and the domain specified by this configuration

    c) “*fs.azure.default.identity.transformer.service.principal.substitution.list*”
– a comma separated list of names to be replaced with the service principal ID specified
by “*fs.azure.default.identity.transformer.service.principal.id*”.  This substitution
occurs when setOwner, setAcl, modifyAclEntries, or removeAclEntries are invoked with identities
contained in the substitution list.

    d) “*fs.azure.default.identity.transformer.service.principal.id*” – an Azure Active
Directory object ID used as the replacement for names contained in the list specified by “*fs.azure.default.identity.transformer.service.principal.substitution.list*”.

    e) The patch 004 properties defined in ConfigurationKeys should be replaced with the
above.  Please include descriptions.

8) Remove the class org.apache.hadoop.fs.azurebfs.oauth.IdentityTransformer.

9) Remove DEFAULT_ENABLE_SHORT_NAME from FileSystemConfigurations.  It can be moved to DefaultIdentityTransformer
if it is needed.

10) I don’t see any reason for IdentityTransformerException to exist, since it does not
have any implementation.  If it truly cannot be removed, I would rename it to *IdentityTransformerInitializationException*.

11) Update all the places in AzureBlobFileSystemStore where the CustomIdentityTransformer
instance is invoked to check for null, and only perform the transformation if non-null.

12) By default, there will be no transformation. A customer can set fs.azure.identity.transformer.type
to the default or a custom implementation as needed.

13) We should have tests for the implementation of DefaultIdentityTransformer.


> ABFS: Enable owner and group conversion for MSI and login user using OAuth
> --------------------------------------------------------------------------
>                 Key: HADOOP-15954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15954
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2.0
>            Reporter: junhua gu
>            Assignee: Da Zhou
>            Priority: Major
>         Attachments: HADOOP-15954-001.patch, HADOOP-15954-002.patch, HADOOP-15954-003.patch,
> Add support for overwriting owner and group in set/get operations to be the service principal
id when OAuth is used. Add support for upn short name format.
> Add Standard Transformer for SharedKey / Service 
> Add interface provides an extensible model for customizing the acquisition of Identity

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message