hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esfandiar Manii (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
Date Tue, 15 Aug 2017 23:19:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128044#comment-16128044

Esfandiar Manii commented on HADOOP-14583:

This is not related to if parent directory does not exist. The issue is concurrency. If there
are multiple threads trying to create and delete the same file over and over, you will be
having a scenario where:
ThreadA -> Creates File
ThreadB -> Looks up the file and it exists
ThreadA -> Removes file
ThreadB -> Looks up for metadata before creation and throws exception

The logic to make the metadata lookup safe was missed from the create function where most
of the other ones have this logic. I updated the code and added a test to ensure this wont
happen under over 100 threads.

Please take a look over the patch and let me know if you have comments.

> wasb throws an exception if you try to create a file and there's no parent directory
> ------------------------------------------------------------------------------------
>                 Key: HADOOP-14583
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14583
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-14583-001.patch
> It's a known defect of the Hadoop FS API (and one we don't explicitly test for enough),
but you can create a file on a path which doesn't exist. In that situation, the create() logic
is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}} without calling
{{mkdirs(filepath.getParent()}} first. That's the semantics expected of {{createNonRecursive()}}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message