hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12897) Path not found when we get the ec policy for a .snapshot dir
Date Tue, 02 Jan 2018 20:56:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308695#comment-16308695
] 

Xiao Chen commented on HDFS-12897:
----------------------------------

Thanks a lot [~GeLiXin] for revving!

Sorry didn't find this earlier. But poking at it, I found another issue that's not correct
with snapshots: snapshots are supposed to be immutable, so in the following sequence of events
for the same dir happens:
# set ec policy to RS(6,3)
# take snapshot s1
# set ec policy to RS(3,2)
# take snapshot s2
{{getPolicy}} on s1 should say RS(6,3), and on s2 should say RS(3,2). 

I think this 1-liner should be able to fix EC policy:
{code:title=FSDirErasureCodingOp#getErasureCodingPolicyForPath}
-        final XAttrFeature xaf = inode.getXAttrFeature();
+        final XAttrFeature xaf = inode.getXAttrFeature(iip.getPathSnapshotId());
{code}
This is not exactly related to the 'path not found' error, but I suggest we fix that while
we're at it. (StoragePolicy isn't correct on this either, can do that as a follow-on)

Some minor comments:
- Suggest we use {{src}} instead of {{iip.getPath()}} as the {{FileNotFoundException}} message,
as throwing the original input would be more obvious to the caller. This also saves the possible
path construction.
- In {{TestErasureCodingPolicies}}, can just use {{fs.mkdirs(ecDir);}} instead of {{mkdir}}
with the default perm.
- Please add a test of the new scenario mentioned above.
- Feels to me we can merge {{testErasureCodingPolicyOnReservedDir}} and {{testErasureCodingPolicyOnReservedRawDir}},
to just verify /.reserved at the beginning.
- Change the comments {{verify the EC policy is null other than an exception}} to {{verify
the EC policy is null, not an exception}} (['other than'|https://www.merriam-webster.com/dictionary/other%20than]
isn't right here)
- {{FSDirErasureCodingOp#getErasureCodingPolicy}}, javadocs change could be updated. 'if it
could not be set' is confusing on a get api. I'm fine if we either 1) explaining that null
could be due to no policy, or replication, or 2) leave it as-is.

> Path not found when we get the ec policy for a .snapshot dir
> ------------------------------------------------------------
>
>                 Key: HDFS-12897
>                 URL: https://issues.apache.org/jira/browse/HDFS-12897
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: erasure-coding, hdfs, snapshots
>    Affects Versions: 3.0.0-alpha1, 3.1.0
>            Reporter: Harshakiran Reddy
>            Assignee: LiXin Ge
>         Attachments: HDFS-12897.001.patch, HDFS-12897.002.patch, HDFS-12897.003.patch
>
>
> Scenario:-
> ---------------
> Operation on snapshot dir.
> *EC policy*
> bin> ./hdfs ec -getPolicy -path /dir/
> RS-3-2-1024k
> bin> ./hdfs ec -getPolicy -path /dir/.snapshot/
> {{FileNotFoundException: Path not found: /dir/.snapshot}}
> bin> ./hdfs dfs -ls /dir/.snapshot/
> Found 2 items
> drwxr-xr-x   - user group          0 2017-12-05 12:27 /dir/.snapshot/s1
> drwxr-xr-x   - user group          0 2017-12-05 12:28 /dir/.snapshot/s2
> *Storagepolicies*
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/.snapshot/
> {{The storage policy of /dir/.snapshot/ is unspecified}}
> bin> ./hdfs storagepolicies -getStoragePolicy -path /dir/
> The storage policy of /dir/:
> BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
> *Which is the correct behavior ?*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message