hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
Date Tue, 08 Nov 2016 02:19:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646188#comment-15646188
] 

Mingliang Liu edited comment on HADOOP-13650 at 11/8/16 2:19 AM:
-----------------------------------------------------------------

{{S3AFileSystem#initialize(name, conf)}}, name may have the user:pass login info for creating
the credentials of AWS client. Also the S3AFS is used to get the bucket name and region. This
may bring extra work if we only support initializing {{DDBMetadataStore}} via Configuration
parameter only. Let's figure out this later; by now let's rely on the S3AFS to initialize
the DDBMetadataStore.

# {{s3a s3guard init -m dynamodb:// s3a://bucket/path}} will use the bucket name as the table
name
# {{s3a s3guard init -m dynamodb://mytable s3a://bucket/path}}; we will set the config key
"fs.s3a.s3guard.ddb.table" in the configuration before initializing the S3AFS. The DDBMetadataStore
will prefer this config if provided to the default bucket name.

Either way, the fully qualified S3A uri/name is needed before we can initialize the MetadataStore
from configuration.

Thanks,


was (Author: liuml07):
{{S3AFileSystem#initialize(name, conf)}}, name may have the user:pass login info for creating
the credentials of AWS client. Also the S3AFS is used to get the bucket name and region. This
may bring extra work if we only support initializing {{DDBMetadataStore}} via Configuration
parameter only. Let's figure out this later; by now let's rely on the S3AFS to initialize
the DDBMetadataStore.

> S3Guard: Provide command line tools to manipulate metadata store.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-13650
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13650
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-13650-HADOOP-13345.000.patch, HADOOP-13650-HADOOP-13345.001.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata store, i.e.,
create or delete metadata store, or {{import}}, {{sync}} the file metadata between metadata
store and S3. 
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message