hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
Date Sat, 14 Jan 2017 00:28:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15822564#comment-15822564
] 

Aaron Fabbri commented on HADOOP-13650:
---------------------------------------

Thanks for the followup patch [~eddyxu].  +1 on the code review.  I will try to get some testing
in this evening or early next week.
{quote}
AF> S3A's listFiles discovers non-empty directories
Thanks for catching this. The comments are outdated now. Since LocatedFileStatus erased the
isEmptyDir, the code here is still valid I think. I modified the comments.
{quote}

Ah.. Another reason the isEmptyDirectory bit should probably be ignored by MetadataStore.
 This will get addressed in HADOOP-13914, so we're good here.

{quote}
 AF> Should we add to dirCache here?
dirCache is used in putParentsIfNotPresent(child); after this statement.
{quote}

Understood, you put the *parent* in the dirCache there.  In this code though, you are putting
the "child" dir in MS, so you could also remember that the child dir is already in MS.  This
current code might put the "child" dir in MS two times (once here and again when you add its
children), depending on iteration order of listFiles().  This does not affect correctness
(it is just perf optimization), so I'm still +1 on this patch.

> S3Guard: Provide command line tools to manipulate metadata store.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-13650
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13650
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-13650-HADOOP-13345.000.patch, HADOOP-13650-HADOOP-13345.001.patch,
HADOOP-13650-HADOOP-13345.002.patch, HADOOP-13650-HADOOP-13345.003.patch, HADOOP-13650-HADOOP-13345.004.patch,
HADOOP-13650-HADOOP-13345.005.patch, HADOOP-13650-HADOOP-13345.006.patch, HADOOP-13650-HADOOP-13345.007.patch,
HADOOP-13650-HADOOP-13345.008.patch, HADOOP-13650-HADOOP-13345.009.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata store, i.e.,
create or delete metadata store, or {{import}}, {{sync}} the file metadata between metadata
store and S3. 
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message