hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13998) initial s3guard preview
Date Thu, 02 Feb 2017 08:46:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849666#comment-15849666

Aaron Fabbri commented on HADOOP-13998:

All of the dependencies for this have patches available for review.  I would like to start
on empty directory handling improvements, but would prefer to merge the feature branch to
trunk first to avoid having to maintain more S3AFileSystem diffs.

*I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent JIRAs linked
here committed.  I'll provide a summary of where we are at below.  I look forward to feedback
from [~stevel@apache.org], [~cnauroth], [~eddyxu], [~mackrorysd], and the rest of the community.

The main feature we want for the initial version is listing consistency, and we've accomplished

For testing, we have completed (off the top of my head):
- List consistency tests with failure injection.  (HADOOP-13793) This integration test forces
a delay in visibility of certain files by wrapping the AWS S3 client. It asserts listing is
consistent.  The test fails without S3Guard, and succeeds with it. 
- All existing S3 integration tests with and without S3Guard.  The filesystem contract tests
have been invaluable here. (HADOOP-13589 makes these very easy to run).
- MetadataStore contract tests that ensure that the API semantics of the DynamoDB and in-memory
reference implementations are correct.
- MetadataStore scale tests that can be used to force DynamoDB service throttling and ensure
we are robust to that.
- Unit tests for different parts of the S3Guard logic.

In addition to this upstream testing, my colleagues have run a couple of our in-house test
harnesses against S3Guard.  This includes Hive, Spark, and a number of other components. 
All the testing is looking great so far.

> initial s3guard preview
> -----------------------
>                 Key: HADOOP-13998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13998
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
> JIRA to link in all the things we think are needed for a preview/merge into trunk

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message