hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
Date Wed, 30 Nov 2016 23:38:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710216#comment-15710216
] 

Aaron Fabbri commented on HADOOP-13793:
---------------------------------------

All s3a integration tests passed with default endpoint in US West 2.

Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractGetFileStatus
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractMkdir
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractSeek
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContext
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextCreateMkdir
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextStatistics
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI
Running org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextUtil
Running org.apache.hadoop.fs.s3a.ITestBlockingThreadPoolExecutorService
Running org.apache.hadoop.fs.s3a.ITestS3AAWSCredentialsProvider
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputArray
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputByteBuffer
Running org.apache.hadoop.fs.s3a.ITestS3ABlockOutputDisk
Running org.apache.hadoop.fs.s3a.ITestS3ABlocksize
Running org.apache.hadoop.fs.s3a.ITestS3AConfiguration
Running org.apache.hadoop.fs.s3a.ITestS3ACredentialsInURL
Running org.apache.hadoop.fs.s3a.ITestS3AEncryption
Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmPropagation
Running org.apache.hadoop.fs.s3a.ITestS3AEncryptionBlockOutputStream
Running org.apache.hadoop.fs.s3a.ITestS3AFailureHandling
Running org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost
Running org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
Running org.apache.hadoop.fs.s3a.ITestS3AMiscOperations
Running org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials
Running org.apache.hadoop.fs.s3a.ITestS3ATestUtils
Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteFilesOneByOne
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles
Running org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesArrayBlocks
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesByteBufferBlocks
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesClassicOutput
Running org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesDiskBlocks
Running org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance
Running org.apache.hadoop.fs.s3a.yarn.ITestS3A
Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster
Running org.apache.hadoop.fs.s3native.ITestInMemoryNativeS3FileSystemContract
...
Failed tests:
  ITestJets3tNativeS3FileSystemContract>NativeS3FileSystemContractBaseTest.testListStatusForRoot:66
Root directory is not empty;  expected:<0> but was:<4>
Tests in error:
  ITestJets3tNativeS3FileSystemContract>FileSystemContractBaseTest.testLSRootDir:727->FileSystemContractBaseTest.assertListFilesFinds:742
ยป FileNotFound
Tests run: 492, Failures: 1, Errors: 1, Skipped: 103

> s3guard: add inconsistency injection, integration tests
> -------------------------------------------------------
>
>                 Key: HADOOP-13793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13793
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13793-HADOOP-13345.001.patch, HADOOP-13793-HADOOP-13345.002.patch
>
>
> Many of us share concerns that testing the consistency features of S3Guard will be difficult
if we depend on the rare and unpredictable occurrence of actual inconsistency in S3 to exercise
those code paths.
> I think we should have a mechanism for injecting failure to force exercising of the consistency
codepaths in S3Guard.
> Requirements:
> - Integration tests that cause S3A to see the types of inconsistency we address with
S3Guard.
> - These are deterministic integration tests.
> Unit tests are possible as well, if we were to stub out the S3Client.  That may be less
bang for the buck, though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message