hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
Date Sat, 18 Feb 2017 02:53:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15872936#comment-15872936

Mingliang Liu commented on HADOOP-13345:

Thanks [~fabbri] for prompt reviewing test report!

Thats ok.. It does miss ITestS3Guard\{ListConsistency, ToolDynamoDB\}
That's a good catch. I just learned these two tests. However, {{org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency}}
fails before/after merge. Do I need to configure something special?
cmvn -Dit.test='ITestS3Guard*' -Dtest=none -Dscale -Ds3guard -Ddynamo -q clean verify

 T E S T S
Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.992 sec <<< FAILURE!
- in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)  Time elapsed:
13.552 sec  <<< FAILURE!
java.lang.AssertionError: Unexpected number of results from metastore. Metastore should only
know about /XYZ: DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack, listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx;
isSymlink=false} isEmptyDirectory=true}, s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123;
isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx;
isSymlink=false} isEmptyDirectory=true}}, isAuthoritative=false}
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127)

Curious, what is our difference in HADOOP-13345 that changes this? Is our feature branch exception
behavior different?
We don't really change anything in that part. I guess the reason is that, when enabling S3Guard,
the code path that fails in S3AFileSystem changes for that test somehow. For example (to be
confirmed), the request w/o S3Guard was calling {{getFileStatus()}} and fails with access
denied exception containing "Forbidden" keyword; while the request w/ S3Guard is able to call
{{getFileStatus()}} and fails later with read operations, which then fails with access denied
exception containing "Access Denied" keyword. So I think relaxing exception message assertion
in test should work just fine.

> S3Guard: Improved Consistency for S3A
> -------------------------------------
>                 Key: HADOOP-13345
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13345
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13345.prototype1.patch, s3c.001.patch, S3C-ConsistentListingonS3-Design.pdf,
S3GuardImprovedConsistencyforS3A.pdf, S3GuardImprovedConsistencyforS3AV2.pdf
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a stronger
consistency model than what is currently offered.  The solution coordinates with a strongly
consistent external store to resolve inconsistencies caused by the S3 eventual consistency

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message