Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 2 Jun 2016 09:35:59 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12974913.1464809826000.12835.1464860159260@Atlassian.JIRA>
In-Reply-To: <JIRA.12974913.1464809826000@Atlassian.JIRA>
References: <JIRA.12974913.1464809826000@Atlassian.JIRA> <JIRA.12974913.1464809826865@arcas>
Subject: [jira] [Commented] (HADOOP-13230) s3a's use of fake empty directory
 blobs does not interoperate with other s3 tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Jun 2016 09:36:01 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312025#comment-15312025 ] 

Steve Loughran commented on HADOOP-13230:
-----------------------------------------

-actually, a {{touch}} into the dest dir should be sufficient to trigger a cleanup, at least until some optimisations go in.

What may be viable here is actually to add some specific cleanup entry point/command to hadoop-aws, which cleans a bucket out of spurious false empty directories. After something starts mixing s3a and raw aws cli calls, this could be used to purge fake directories up the tree

> s3a's use of fake empty directory blobs does not interoperate with other s3 tools
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-13230
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13230
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Aaron Fabbri
>
> Users of s3a may not realize that, in some cases, it does not interoperate well with other s3 tools, such as the AWS CLI.  (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote in the first step, causes s3a (listStatus() etc.) to continue to treat that directory as empty, even though the second step was supposed to populate the directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by design".. May also be interesting to brainstorm solutions and/or a config option to change the behavior if folks care.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org