hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
Date Tue, 01 Nov 2016 23:08:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626982#comment-15626982

Mingliang Liu commented on HADOOP-13650:

Thanks for the nice work, [~eddyxu]!

One question I have in the implementation is that, for initialize() / destroy functions, can
we provide a version of such functions that do not take S3FIleSystem as parameters (i.e.,
taking Configuration instead)? Providing s3a path in s3a s3guard init -m <URI> s3a://bucket/path
seems be not necessary.
Basically we can create a S3AFileSystem via URI (bucket) and conf; and I thought we could
initialize the s3a fs object anywhere.

The DDBMetadataStore needs the region, bucket (default table name), AWS keys etc, among which
the bucket is not specified in the configuration. Newly added config key DDB_TABLE_NAME_KEY
can be helpful in this case. In short, I think it's possible to achieve this. Pssing s3a URI
along with the MetadataStore URI seems not necessary to me either in the CLI tool. You can
assume I can construct a DDBMetadataStore using configuration simply. I'll let you know if
I can't. This will be considered in the [HADOOP-13449] v2 patch, on its way.

I'll come back to review this patch this week.

> S3Guard: Provide command line tools to manipulate metadata store.
> -----------------------------------------------------------------
>                 Key: HADOOP-13650
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13650
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-13650-HADOOP-13345.000.patch
> Similar systems like EMRFS has the CLI tools to manipulate the metadata store, i.e.,
create or delete metadata store, or {{import}}, {{sync}} the file metadata between metadata
store and S3. 
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message