Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 13 Apr 2017 18:15:42 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13060836.1491005114000.278180.1492107342146@Atlassian.JIRA>
In-Reply-To: <JIRA.13060836.1491005114000@Atlassian.JIRA>
References: <JIRA.13060836.1491005114000@Atlassian.JIRA> <JIRA.13060836.1491005114307@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14266) S3Guard:
 S3AFileSystem::listFiles() to employ MetadataStore
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 13 Apr 2017 18:15:48 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967989#comment-15967989 ] 

Steve Loughran commented on HADOOP-14266:
-----------------------------------------

Let's wait for Aaron & his review.

* Never noticed the code style {{for (; statusIterator.hasNext();)}}. before. Nice - I'll use it myself
* added a patch with some javadoc tweaks, and made the empty FileStatus entry returned on all non-s3guard listings a constant.
* testing, s3 ireland.
* got one test failure on the 004 patch, with    args {{-Dparallel-tests -DtestsThreadCount=8 -Ddynamo}}:

{code}
Tests run: 44, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.082 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract
testRenameEmptyToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)  Time elapsed: 3.221 sec  <<< FAILURE!
junit.framework.AssertionFailedError: Rename result expected:<true> but was:<false>
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:174)
	at junit.framework.TestCase.assertEquals(TestCase.java:333)
	at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:567)
	at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameEmptyToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:715)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at junit.framework.TestCase.runBare(TestCase.java:141)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:129)
	at junit.framework.TestSuite.runTest(TestSuite.java:255)
	at junit.framework.TestSuite.run(TestSuite.java:250)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Tests run: 61, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 164.347 sec - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations

Results :

Failed tests: 
  ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameEmptyToDirWithSamePrefixAllowed:715->FileSystemContractBaseTest.rename:567 Rename result expected:<true> but was:<false>

{code}

This failure implies that attempting to rename an empty dir to itself failed, when it should be a no-op returning true. Maybe the dir wasn't considered empty. I could not create this when running 

I've managed to get this, with my changes, and after rolling back, with/without dynamo.
{code}
Tests run: 4, Failures: 0, Errors: 1, Skipped: 3, Time elapsed: 12.364 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency
testConsistentListFiles(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)  Time elapsed: 8.944 sec  <<< ERROR!
java.io.FileNotFoundException: No such file or directory: s3a://hwdev-steve-ireland-new/fork-6/test/doTestListFiles-0-0-1-true
	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1852)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1748)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1710)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListFiles(S3AFileSystem.java:2435)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.listFiles(S3AFileSystem.java:2418)
	at org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.doTestListFiles(ITestS3GuardListConsistency.java:231)
	at org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testConsistentListFiles(ITestS3GuardListConsistency.java:170)
{code}


> S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
> -----------------------------------------------------------
>
>                 Key: HADOOP-14266
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14266
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-14266-HADOOP-13345.000.patch, HADOOP-14266-HADOOP-13345.001.patch, HADOOP-14266-HADOOP-13345.002.patch, HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch, HADOOP-14266-HADOOP-13345.005.patch
>
>
> Similar to [HADOOP-13926], this is to track the effort of employing MetadataStore in {{S3AFileSystem::listFiles()}}.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org