Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 92BB818282 for ; Tue, 30 Jun 2015 19:01:16 +0000 (UTC) Received: (qmail 72658 invoked by uid 500); 30 Jun 2015 19:01:15 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 72367 invoked by uid 500); 30 Jun 2015 19:01:14 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 71778 invoked by uid 99); 30 Jun 2015 19:01:14 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2015 19:01:14 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 59421E1091; Tue, 30 Jun 2015 19:01:14 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: aw@apache.org To: common-commits@hadoop.apache.org Date: Tue, 30 Jun 2015 19:01:16 -0000 Message-Id: In-Reply-To: <15686b9e51d04661aafff0db57058be8@git.apache.org> References: <15686b9e51d04661aafff0db57058be8@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [03/27] hadoop git commit: HADOOP-12009 Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. (J.Andreina via stevel) HADOOP-12009 Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus. (J.Andreina via stevel) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3dfa8161 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3dfa8161 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3dfa8161 Branch: refs/heads/HADOOP-12111 Commit: 3dfa8161f9412bcb040f3c29c471344f25f24337 Parents: b543d1a Author: Steve Loughran Authored: Sun Jun 28 19:13:48 2015 +0100 Committer: Steve Loughran Committed: Sun Jun 28 19:14:00 2015 +0100 ---------------------------------------------------------------------- hadoop-common-project/hadoop-common/CHANGES.txt | 4 ++++ .../main/java/org/apache/hadoop/fs/FileSystem.java | 17 ++++++++++++++++- .../src/site/markdown/filesystem/filesystem.md | 4 ++++ .../hadoop/fs/FileSystemContractBaseTest.java | 11 ++++++++--- 4 files changed, 32 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/3dfa8161/hadoop-common-project/hadoop-common/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt index 18c73c0..132e4c0 100644 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ b/hadoop-common-project/hadoop-common/CHANGES.txt @@ -661,6 +661,10 @@ Release 2.8.0 - UNRELEASED HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error occurs (Jason Lowe via jeagles) + + HADOOP-12009 Clarify FileSystem.listStatus() sorting order & fix + FileSystemContractBaseTest:testListStatus. (J.Andreina via stevel) + OPTIMIZATIONS HADOOP-11785. Reduce the number of listStatus operation in distcp http://git-wip-us.apache.org/repos/asf/hadoop/blob/3dfa8161/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java index 01d4b27..3f9e3bd 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java @@ -1498,7 +1498,9 @@ public abstract class FileSystem extends Configured implements Closeable { /** * List the statuses of the files/directories in the given path if the path is * a directory. - * + *

+ * Does not guarantee to return the List of files/directories status in a + * sorted order. * @param f given path * @return the statuses of the files/directories in the given patch * @throws FileNotFoundException when the path does not exist; @@ -1540,6 +1542,9 @@ public abstract class FileSystem extends Configured implements Closeable { /** * Filter files/directories in the given path using the user-supplied path * filter. + *

+ * Does not guarantee to return the List of files/directories status in a + * sorted order. * * @param f * a path name @@ -1560,6 +1565,9 @@ public abstract class FileSystem extends Configured implements Closeable { /** * Filter files/directories in the given list of paths using default * path filter. + *

+ * Does not guarantee to return the List of files/directories status in a + * sorted order. * * @param files * a list of paths @@ -1576,6 +1584,9 @@ public abstract class FileSystem extends Configured implements Closeable { /** * Filter files/directories in the given list of paths using user-supplied * path filter. + *

+ * Does not guarantee to return the List of files/directories status in a + * sorted order. * * @param files * a list of paths @@ -1736,6 +1747,8 @@ public abstract class FileSystem extends Configured implements Closeable { * while consuming the entries. Each file system implementation should * override this method and provide a more efficient implementation, if * possible. + * Does not guarantee to return the iterator that traverses statuses + * of the files in a sorted order. * * @param p target path * @return remote iterator @@ -1763,6 +1776,8 @@ public abstract class FileSystem extends Configured implements Closeable { /** * List the statuses and block locations of the files in the given path. + * Does not guarantee to return the iterator that traverses statuses + * of the files in a sorted order. * * If the path is a directory, * if recursive is false, returns files in the directory; http://git-wip-us.apache.org/repos/asf/hadoop/blob/3dfa8161/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md index 84e3755..f323374 100644 --- a/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md +++ b/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md @@ -183,6 +183,10 @@ to the same path: forall fs in listStatus(Path) : fs == getFileStatus(fs.path) +**Ordering of results**: there is no guarantee of ordering of the listed entries. +While HDFS currently returns an alphanumerically sorted list, neither the Posix `readdir()` +nor Java's `File.listFiles()` API calls define any ordering of returned values. Applications +which require a uniform sort order on the results must perform the sorting themselves. ### Atomicity and Consistency http://git-wip-us.apache.org/repos/asf/hadoop/blob/3dfa8161/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java index 2ca81e9..495af00 100644 --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java @@ -20,6 +20,7 @@ package org.apache.hadoop.fs; import java.io.FileNotFoundException; import java.io.IOException; +import java.util.ArrayList; import junit.framework.TestCase; @@ -224,9 +225,13 @@ public abstract class FileSystemContractBaseTest extends TestCase { paths = fs.listStatus(path("/test/hadoop")); assertEquals(3, paths.length); - assertEquals(path("/test/hadoop/a"), paths[0].getPath()); - assertEquals(path("/test/hadoop/b"), paths[1].getPath()); - assertEquals(path("/test/hadoop/c"), paths[2].getPath()); + ArrayList list = new ArrayList(); + for (FileStatus fileState : paths) { + list.add(fileState.getPath().toString()); + } + assertTrue(list.contains(path("/test/hadoop/a"))); + assertTrue(list.contains(path("/test/hadoop/b"))); + assertTrue(list.contains(path("/test/hadoop/c"))); paths = fs.listStatus(path("/test/hadoop/a")); assertEquals(0, paths.length);