Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4B6E818726 for ; Thu, 2 Jul 2015 09:08:05 +0000 (UTC) Received: (qmail 69539 invoked by uid 500); 2 Jul 2015 09:08:05 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 69485 invoked by uid 500); 2 Jul 2015 09:08:05 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 69472 invoked by uid 99); 2 Jul 2015 09:08:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2015 09:08:05 +0000 Date: Thu, 2 Jul 2015 09:08:04 +0000 (UTC) From: "Pieter Reuse (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-12169) ListStatus on empty dir in S3A lists itself instead of returning an empty list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611682#comment-14611682 ] Pieter Reuse commented on HADOOP-12169: --------------------------------------- Yes, this bugfix is intended for Hadoop version 2.7. The bug was introduced in Hadoop-2.6 and Hadoop-2.7 is affected by it. > ListStatus on empty dir in S3A lists itself instead of returning an empty list > ------------------------------------------------------------------------------ > > Key: HADOOP-12169 > URL: https://issues.apache.org/jira/browse/HADOOP-12169 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.6.0, 2.7.0, 2.7.1 > Reporter: Pieter Reuse > Assignee: Pieter Reuse > Attachments: HADOOP-12169-001.patch > > > Upon testing the patch for HADOOP-11918, I stumbled upon a weird behaviour this introduces to the S3AFileSystem-class. Calling ListStatus() on an empty bucket returns an empty list, while doing the same on an empty directory, returns an array of length 1 containing only this directory itself. > The bugfix is quite simple. In the line of code {code}...if (keyPath.equals(f)...{code} (S3AFileSystem:758), keyPath is qualified wrt. the fs and f is not. Therefore, this returns false while it shouldn't. The bugfix to make f qualified in this line of code. > More formally: accoring to the formal definition of [The Hadoop FileSystem API Definition|https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/filesystem/], more specifically FileSystem.listStatus, only child elements of a directory should be returned upon a listStatus()-call. > In detail: > {code} > elif isDir(FS, p): result [getFileStatus(c) for c in children(FS, p) where f(c) == True] > {code} > and > {code} > def children(FS, p) = {q for q in paths(FS) where parent(q) == p} > {code} > Which translates to the result of listStatus on an empty directory being an empty list. This is the same behaviour as ls has in Unix, which is what someone would expect from a FileSystem. > Note: it seemed appropriate to add the test of this patch to the same file as the test for HADOOP-11918, but as a result, one of the two will have to be rebased wrt. the other before being applied to trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)