hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
Date Fri, 20 Jan 2017 11:40:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831590#comment-15831590
] 

Steve Loughran commented on HADOOP-13876:
-----------------------------------------

That's going to be a separate topic. Looking at this patch, there's a new property doc in
s3guard.md (which is good), and changes to two tests —changes which aren't needed

We can guarantee that s3guard is disabled by explicitly disabling in test/resources/core-site.xml.
This is what I think we should do, looking at my auth-keys.xml file (indeed, making me suspect
that the csv files are only working because of the .impl property).

Why not force it in the tests? Because the reason for making the CSV URL configurable is so
that people testing against their own S3 infra (e.g the WDC team) have the option to point
to a local file. Now, they don't need s3guard, having a consistent store, but it may be that
others (me?) want to test against a different file with s3guard turned on, just to see what
happens.

I think we could just add these values to the test core-site and rely on it to set things
up...people can override if they want

{code}

  <property>
    <name>fs.s3a.bucket.landsat-pds.endpoint</name>
    <value>${central.endpoint}</value>
    <description>The endpoint for s3a://landsat-pds URLs</description>
  </property>

  <property>
    <name>fs.s3a.bucket.landsat-pds.metadatastore.impl</name>
    <value>${s3guard.null}</value>
    <description>The read-only landsat-pds repository isn't
    managed by s3guard</description>
  </property>

  <property>
    <name>s3guard.null</name>
    <value>org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore</value>
  </property>

  <property>
    <name>s3guard.dynamo</name>
    <value>org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore</value>
  </property>
  <property>
    <name>frankfurt.endpoint</name>
    <value>s3.eu-central-1.amazonaws.com</value>
  </property>

  <property>
    <name>ireland.endpoint</name>
    <value>s3-eu-west-1.amazonaws.com</value>
  </property>

  <property>
    <name>central.endpoint</name>
    <value>s3.amazonaws.com</value>
  </property>
{code}

> S3Guard: better support for multi-bucket access including read-only
> -------------------------------------------------------------------
>
>                 Key: HADOOP-13876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13876
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13876-HADOOP-13345.000.patch
>
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options for choosing DynamoDB table names:
> 1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
> 2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
> One of the issues is with accessing read-only buckets.  If a user accesses a read-only
bucket with credentials that do not have DynamoDB write permissions, they will get errors
when trying to access the read-only bucket.  This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the real use-case.
> - Allow for a "one DynamoDB table per cluster" configuration with a way to chose which
credentials are used for DynamoDB.
> - Document limitations etc. in the s3guard.md site doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message