hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13230) s3a's use of fake empty directory blobs does not interoperate with other s3 tools
Date Thu, 02 Jun 2016 09:35:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312025#comment-15312025

Steve Loughran commented on HADOOP-13230:

-actually, a {{touch}} into the dest dir should be sufficient to trigger a cleanup, at least
until some optimisations go in.

What may be viable here is actually to add some specific cleanup entry point/command to hadoop-aws,
which cleans a bucket out of spurious false empty directories. After something starts mixing
s3a and raw aws cli calls, this could be used to purge fake directories up the tree

> s3a's use of fake empty directory blobs does not interoperate with other s3 tools
> ---------------------------------------------------------------------------------
>                 Key: HADOOP-13230
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13230
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Aaron Fabbri
> Users of s3a may not realize that, in some cases, it does not interoperate well with
other s3 tools, such as the AWS CLI.  (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote in the
first step, causes s3a (listStatus() etc.) to continue to treat that directory as empty, even
though the second step was supposed to populate the directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by design"..
May also be interesting to brainstorm solutions and/or a config option to change the behavior
if folks care.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message