hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14124) S3AFileSystem silently deletes "fake" directories when writing a file.
Date Fri, 28 Apr 2017 10:01:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988565#comment-15988565

Steve Loughran commented on HADOOP-14124:

revisiting this. S3A FS should recreate a fake directory {{/job/task}}, when {{job/task/file}}
is deleted, and there were no other entries under {{job/task}}. Look at the method {{S3AFileSystem.createFakeDirectoryIfNecessary()}}

> S3AFileSystem silently deletes "fake" directories when writing a file.
> ----------------------------------------------------------------------
>                 Key: HADOOP-14124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Joel Baranick
>              Labels: filesystem, s3
> I realize that you guys probably have a good reason for {{S3AFileSystem}} to cleanup
"fake" folders when a file is written to S3.  That said, that fact that it silently does this
feels like a separation of concerns issue.  It also leads to weird behavior issues where calls
to {{AmazonS3Client.getObjectMetadata}} for folders work before calling {{S3AFileSystem.create}}
but not after.  Also, there seems to be no mention in the javadoc that the {{deleteUnnecessaryFakeDirectories}}
method is automatically invoked. Lastly, it seems like the goal of {{FileSystem}} should be
to ensure that code built on top of it is portable to different implementations.  This behavior
is an example of a case where this can break down.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message