hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-14457) LocalMetadataStore falsely reports empty parent directory authoritatively
Date Thu, 25 May 2017 16:04:04 GMT
Sean Mackrory created HADOOP-14457:
--------------------------------------

             Summary: LocalMetadataStore falsely reports empty parent directory authoritatively
                 Key: HADOOP-14457
                 URL: https://issues.apache.org/jira/browse/HADOOP-14457
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Sean Mackrory


Not a great test yet, but it at least reliably demonstrates the issue. LocalMetadataStore
will sometimes erroneously report that a directory is empty with isAuthoritative = true when
it *definitely* has children the metadatastore should know about. It doesn't appear to happen
if the children are just directory. The fact that it's returning an empty listing is concerning,
but the fact that it says it's authoritative *might* be a second bug.

{code}
diff --git a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index 78b3970..1821d19 100644
--- a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++ b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -965,7 +965,7 @@ public boolean hasMetadataStore() {
   }
 
   @VisibleForTesting
-  MetadataStore getMetadataStore() {
+  public MetadataStore getMetadataStore() {
     return metadataStore;
   }
 
diff --git a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
index 4339649..881bdc9 100644
--- a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
+++ b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
@@ -23,6 +23,11 @@
 import org.apache.hadoop.fs.contract.AbstractFSContract;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.Tristate;
+import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata;
+import org.apache.hadoop.fs.s3a.s3guard.MetadataStore;
+import org.junit.Test;
 
 import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
 import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
@@ -72,4 +77,24 @@ public void testRenameDirIntoExistingDir() throws Throwable {
     boolean rename = fs.rename(srcDir, destDir);
     assertFalse("s3a doesn't support rename to non-empty directory", rename);
   }
+
+  @Test
+  public void testMkdirPopulatesFileAncestors() throws Exception {
+    final FileSystem fs = getFileSystem();
+    final MetadataStore ms = ((S3AFileSystem) fs).getMetadataStore();
+    final Path parent = path("testMkdirPopulatesFileAncestors/source");
+    try {
+      fs.mkdirs(parent);
+      final Path nestedFile = new Path(parent, "dir1/dir2/dir3/file4");
+      byte[] srcDataset = dataset(256, 'a', 'z');
+      writeDataset(fs, nestedFile, srcDataset, srcDataset.length,
+          1024, false);
+
+      DirListingMetadata list = ms.listChildren(parent);
+      assertTrue("MetadataStore falsely reports authoritative empty list",
+          list.isEmpty() == Tristate.FALSE || !list.isAuthoritative());
+    } finally {
+      fs.delete(parent, true);
+    }
+  }
 }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message