hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
Date Tue, 09 Jul 2013 18:57:50 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-9361:
-----------------------------------

    Attachment: HADOOP-9361-003.patch

This is up as a work in progress; if you add the right details to the auth-keys.xml file in
common/src/test/resources it will test S3n and ftp filesystems as well as local; HDFS adds
its tests too.

Some variances
# HDFS won't let you rm / even if empty, or rm -rf /. 
# FTP throws FileNotFoundException if you try to delete a file that isn't there.
# FTP (the back end?) will overwrite a directory with a file creation.
# {{RawLocalFS.rename()}} will try to do a {{File.rename()}} operation and fall back to a
copy, this is the outstanding issue from HDFS-303, "What should consistent rename actions
be"
# S3n will not only let you overwrite a dir with a file (side effect of how blobstores use
0-byte files as dir markers), it will do this even if the destination has children. It could
check for that
                
> Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, test
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets
tested downstream, other filesystems, such as blobstore bindings, don't.
> The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258
shows is incomplete.
> I propose 
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes, instead
of one big test suite. 
> # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename,
atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases
if a feature is missing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message