hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manoj Govindassamy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12203) INodeAttributesProvider#getAttributes() support for default/passthrough mode
Date Wed, 26 Jul 2017 20:00:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102195#comment-16102195

Manoj Govindassamy commented on HDFS-12203:

Adding [~yzhangal], [~jojochuang], [~daryn] for more discussion.

> INodeAttributesProvider#getAttributes() support for default/passthrough mode
> ----------------------------------------------------------------------------
>                 Key: HDFS-12203
>                 URL: https://issues.apache.org/jira/browse/HDFS-12203
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Manoj Govindassamy
>            Assignee: Manoj Govindassamy
> {{INodeAttributeProvider}}
> {noformat}
>   ..
>   public abstract INodeAttributes getAttributes(String[] pathElements, INodeAttributes
>   public INodeAttributes getAttributes(byte[][] components, INodeAttributes inode) {
>     String[] elements = new String[components.length];
>     for (int i = 0; i < elements.length; i++) {
>       elements[i] = DFSUtil.bytes2String(components[i]);
>     }
>     return getAttributes(elements, inode);
>   }
> {noformat}
> The abstract method {{INodeAttributesProvider#getAttributes()}} currently takes in INode
path elements and the INode. Many a times, the provider falls back to the local INode attributes.
But, there isn't a way for the caller to tell the provider to fall back to local attribute
on demand. Below is one such usecase, where the caller of getAttributes() wants to use the
local attributes instead of the one coming from the external provider.
> Assume the external attributes provider takes care of providing custom ACLFeature for
some specific directories and all its descendants (like Sentry) and fallsback to the default
INode ACLFeature for all other files/directories. And lets assume that the external provider
also has its meta data properly replicated to another site. 
> An operation like snapshot diff iterates over the children of given snapshot pair and
generates the diff list. In order to generate the diff list, files are compared which involves
their metadata comparisons. Once HDFS-12201 is fixed, metadata comparisons will compare the
attributes from external providers as well. Given that external attributes can be ignored
as they are taken care by other means, Snapshot diff wants the meta data comparison to ignore
the features coming from external attribute provider and fall back to the local INode features.
 This on-demand fall-back or passthrough mode for INodeAttributesProvider will help to avoid
the huge performance, or memory over consumption problems arising out of the external provider

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message