hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13960) Initialize DynamoDBMetadataStore without associated S3AFileSystem
Date Thu, 12 Jan 2017 13:19:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15820976#comment-15820976

Steve Loughran commented on HADOOP-13960:

this may need extension with the name of the s3a bucket, so that per-bucket options will be
pulled in. otherwise only the base fs.s3a settings are picked up

> Initialize DynamoDBMetadataStore without associated S3AFileSystem
> -----------------------------------------------------------------
>                 Key: HADOOP-13960
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13960
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>             Fix For: HADOOP-13345
>         Attachments: HADOOP-13960-HADOOP-13345.000.patch
> Per the discussion in [HADOOP-13650], it's helpful to initialize a DynamoDBMetadataStore
object without S3AFileSystem. In the current code, we can achieve this via {{DynamoDBMetadataStore#initialize(Configuration)}}.
However, users still have to provide the associated S3AFileSystem URI in the configuration,
by means of either setting the {{fs.defaultFS}} in configuration file or {{-fs s3://bucket}}
command line parameter. Setting the default FileSystem in configuration seems not necessary
as the command line is to manipulate metadata store, e.g. command line tools on an existing
HDFS cluster.
> This JIRA is to track the effort of initializing a DynamoDBMetadataStore without associating
any S3 buckets, so that S3AFileSystem is not needed. Users have to specify in configuration
the DynamoDB endpoints (for region) and table name along with credentials, AWS client settings.
> See [~eddyxu] and [~liuml07]'s comments in [HADOOP-13650] for more details.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message