hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9361) Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
Date Thu, 12 Jun 2014 23:10:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14029976#comment-14029976

Andrew Wang commented on HADOOP-9361:

Hi Steve,

I'm going to take it on faith that you applied all the doc improvements, since honestly there's
too much too look through :) Going to focus on code questions this time around. It's still
hard for me to run the S3 / Swift tests since I'd have to get credentials, but I can do it
if needed.

* Only big feedback I have, is there some way we could have a single parent test for each
FS that runs all its contracts? All the concrete TestFooBarContract classes are basically
copy paste, I'd like to get rid of them. It's also harder for poor me to run all the HDFS
contract tests.

* site.xml, link is still not quite right, should be filesystem/index.html
* FSExceptionMessages, should we re-use the Java messages for these exceptions? I think they're
a bit different. There are also a bunch of places where these strings *aren't* used, but unifying
it everywhere is a tall order. Maybe a follow-on JIRA. Another maybe nice follow-on would
be having a factory method for things like FileAlreadyExistsException where we stitch something
to a string.
* I'd prefer to flip "fs.contract.supports-seek-past-eof" so we can have all trues for HDFS's
contract. Strawman, "supports-exception-on-seek-past-eof"?
* Jets3tnative: typo: maxListingLengthm
* Jets3tnative: there are two handleBlahException methods that just delegate to handleException.
Can we just get rid of them?
* NativeS3, read's LOG.info has an extra "{}"
* Some Contract class javadocs are inconsistent. HDFSContract's class javadoc says "Local
filesystem". SwiftContract mentions S3N.

I think we're pretty close, modulo the big one. Thanks again for the effort here.

> Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-9361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, test
>    Affects Versions: 3.0.0, 2.4.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch, HADOOP-9361-003.patch,
HADOOP-9361-004.patch, HADOOP-9361-005.patch, HADOOP-9361-006.patch, HADOOP-9361-007.patch,
HADOOP-9361-008.patch, HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
HADOOP-9361-013.patch, HADOOP-9361-014.patch, HADOOP-9361-015.patch, HADOOP-9361.awang-addendum.patch
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while HDFS gets
tested downstream, other filesystems, such as blobstore bindings, don't.
> The only tests that are common are those of {{FileSystemContractTestBase}}, which HADOOP-9258
shows is incomplete.
> I propose 
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes, instead
of one big test suite. 
> # Having each FS declare via a properties file what behaviors they offer, such as atomic-rename,
atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases
if a feature is missing.

This message was sent by Atlassian JIRA

View raw message