hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "madhumita chakraborty (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
Date Tue, 23 Feb 2016 20:43:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159609#comment-15159609
] 

madhumita chakraborty commented on HADOOP-12535:
------------------------------------------------

Took care if the comments and uploaded new patch.
[~cnauroth] Some doubts
1.	How do atomic rename and delete settings (fs.contract.supports-atomic-directory-delete
and fs.contract.supports-atomic-directory-rename) get used? I dont see any reference to SUPPORTS_ATOMIC_RENAME
and SUPPORTS_ATOMIC_DIRECTORY_DELETE  defined in ContractOptions. If a filesystem supports
atomic rename, do they need to write new contract test?
2.	I added contract test for append. testRenameFileBeingAppended if failing. I have skipped
this test for now. This test case appends a file and then renames it.

FSDataOutputStream outputStream = getFileSystem().append(target);
      outputStream.write(dataset);
      Path renamed = new Path(testPath,”renamed”);
      Rename(target, renamed);
     outputStream.close();

So here we see that after renaming it closes the original file stream. Was this test written
in this way to maintain parity with unix file system? But WASB does not support this semantic.
If the file is renamed before closing the original stream then we get 2 types of error.
a.	In append we take SelfRenewingLease on the blob. In Close we see if the lease has expired
it will try to renew the lease and that will throw exception as the blob has already been
deleted as part of rename.
b.	In close we commit the blocks of blockblob. So if we rename the file before closing the
stream, actually the content does not get committed. And empty renamed file is created.
cc [~dchickabasapa]

3.Root folder access is not fully supported in WASB. So removed TestAzureNativeContractRootDir.

> Run FileSystem contract tests with hadoop-azure.
> ------------------------------------------------
>
>                 Key: HADOOP-12535
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12535
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: azure, test
>            Reporter: Chris Nauroth
>            Assignee: madhumita chakraborty
>         Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch, HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for hadoop-azure/WASB.
 The contract tests define the expected semantics of the {{FileSystem}}, so running these
for hadoop-azure is likely to catch potential problems and improve overall quality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message