hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
Date Sat, 31 May 2014 12:31:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-9361:

    Status: Open  (was: Patch Available)

looking at the warnings. I think that test failure is a false alarm, but will resubmit to
see; I've also tuned s3n's exception logic a bit -that's a bit of a troublespot.

Andrew, most of this is documentation -what HDFS does and where other filesystems differ within
their "correct" operation, fixes for the other filesystems to make them more consistent with
HDFS and more robust -especially to {{seek()}} and {{close()}}. For HDFS, all that changes
is that out of range seeks are raised as {{EOFException}} instead of simple {{IOException}}
 - with the text unchanged in case people were looking for that in their code.

The thing I'd like input on is actually that specification. Does it make sense -and can you
see obvious mistakes in it. Rename() is the troublespot, incidentally -in the spec, the implementation
and the tests. As for the tests, they've got a more contract-driven architecture, with an
XML file for each FS defining behaviour, e.g [hdfs.xml|https://github.com/steveloughran/hadoop-trunk/blob/stevel/HADOOP-9361-filesystem-contract/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/contract/hdfs.xml].
These are in the test JARs to stop them being used by code, though I'd like them (somehow)
to get down to any functional tests in Bigtop.

Have a look at [that specification|https://github.com/steveloughran/hadoop-trunk/tree/stevel/HADOOP-9361-filesystem-contract/hadoop-common-project/hadoop-common/src/site/markdown/filesystem]
and call out troublespots.

I'll be in the US after hadoop summit if you want to talk on it, though JIRA is where detailed
feedback should go for the historical record.

> Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-9361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, test
>    Affects Versions: 2.4.0, 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch,
HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch,
HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets
tested downstream, other filesystems, such as blobstore bindings, don't.
> The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258
shows is incomplete.
> I propose 
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes, instead
of one big test suite. 
> # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename,
atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases
if a feature is missing.

This message was sent by Atlassian JIRA

View raw message