hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13095) Improve slice tree traversal implementation
Date Thu, 01 Feb 2018 19:35:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349140#comment-16349140

Xiao Chen commented on HDFS-13095:

Thanks for the thoughts Daryn.

Filling in some more info on snapshots: I recently tried on HDFS-13021 but it appears feature
matrix of snapshots with other stuff is complicated and various case-by-case... (no intention
to discuss that one :))

I agree with your take on the problem(s) in general, OTOH there are also cases that's debatable.
For SPS, it gets tricky if the change was the other way round (moving a current file to be
hot with no intention of making all backups hot). For re-encryption, good point. I didn't
realize the fact that it can be (rightfully) considered immutable.

Our decision at that time was: emergency key rolling will have to delete old snapshots for
security; old keys should be kept long enough to allow reading from snapshots. It would be
great if it gets smarter in v2 and relax this dependency...

> Improve slice tree traversal implementation
> -------------------------------------------
>                 Key: HDFS-13095
>                 URL: https://issues.apache.org/jira/browse/HDFS-13095
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>            Priority: Major
> This task is to refine the existing slice tree traversal logic in [ReencryptionHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ReencryptionHandler.java#L74] class.
> Please refer Daryn's review comments
> {quote}*FSTreeTraverser*
>  I need to study this more but I have grave concerns this will work correctly in a mutating
namesystem.  Ex. renames and deletes esp. in combination with snapshots. Looks like there's
a chance it will go off in the weeds when backtracking out of a renamed directory.
> traverseDir may NPE if it's traversing a tree in a snapshot and one of the ancestors
is deleted.
> Not sure why it's bothering to re-check permissions during the crawl.  The storage policy
is inherited by the entire tree, regardless of whether the sub-contents are accessible. 
The effect of this patch is the storage policy is enforced for all readable files, non-readable
violate the new storage policy, new non-readable will conform to the new storage policy. 
Very convoluted.  Since new files will conform, should just process the entire tree.
> {quote}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message