hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jay vyas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10461) Create an extensible single point entry for of HCFS tests - using RawLocalFileSystem as example impl.
Date Wed, 23 Apr 2014 15:28:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978301#comment-13978301

jay vyas commented on HADOOP-10461:

ping [~stevel@apache.org] .... Now that HADOOP-9361 is finalized, do you see value in this
single point of entry for HCFS tests?

> Create an extensible single point entry for of HCFS tests - using RawLocalFileSystem
as example impl.
> -----------------------------------------------------------------------------------------------------
>                 Key: HADOOP-10461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10461
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: jay vyas
>            Priority: Minor
> Currently alot of manual inheritance and stub classes are required in order to run the
FileSystemBaseContract and FSMainOperations tests.   This means that the HCFS Test suite for
any given FileSystem spans many classes, and makes it confusing and cumbersome for anyone
to create new tests for a file system, and also very hard to audit and update the tests for
a particular file system.
> *If each FileSystem had a single point of entry for its tests, we would be much better
> There are two ways to do this.  
> * Have a folder or package for each file system (i.e.  .//hadoop-common/src/test/java/org/apache/hadoop/fs/rawlocal
) Or
> * Wrap all the HCFS Test implementation classes into a single class (i.e. RawLocalFileSystemHCFSTestSuite.java).
> *The purpose of this JIRA is to implement a wrapper class for the RawLocalFileSystem,
which serves as a single point of entry to ALL RawLocalFileSystem tests.  This will cleanup
the RawLocalFileSystem tests while providing other HCFS implementors a template they can use
for their own filesystems (i.e. they can just copy the code and customize it however they
> For example:  This is a sampling of the classes necessary to glue Local FS implementations
into the Existing generic FileSystem test classes: 
> {noformat}
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFcLocalFsUtil.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystemPermission.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFsFCStatistics.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextCreateMkdir.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFSFileContextMainOperations.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFS.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileContext.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/TestSymlinkLocalFSFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithAuthorityLocalFileSystem.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsLocalFs.java
> .//hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithAuthorityLocalFs.java
> {noformat}
> Lets wrap these tests into a Suite for the RawLocalFileSystem, such that any once can
copy that suite to build tests for their own file system. 
> Using the "Enclosed" Junit (http://junit-team.github.io/junit/javadoc/4.10/org/junit/experimental/runners/Enclosed.html
) class, we can combine all of the RawLocalFileSystem tests into a single class.  That way
> * Can save on boiler plate for setup and declaring the same FileSystem stubs over and
over again, making the test "logic" more easy to focus on.  For example, here are some repeat
file context declarations for ViewFS tests:
> {noformat}
> 01:42:08 {TRUNK} $ git grep "ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper)"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcCreateMkdirLocalFs.java:
   fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcMainOperationsLocalFs.java:
   fc = ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestFcPermissionsLocalFs.java:
   return ViewFsTestSetup.setupForViewFsLocalFs(fileContextTestHelper);
> 01:45:19 {TRUNK} $ git grep " FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);"
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsURIs.java:    FileContext.getFileContext(FsConstants.VIEWFS_URI,
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java:    fcView
= FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java:    FileContext
fc = FileContext.getFileContext(FsConstants.VIEWFS_URI, conf);
> {noformat}
> * Can easily be adopted for other FileSystems.
> * Can read in System properties to skip certain tests , thus providing support for the
type of variability that we know FileSystem tests require.  These can then be used to implement
the semantics of HADOOP-9361.
> Ideally, we could replace RawLocalFileSystem tests with this injector as a second follow
up patch to this, it would reduce the overall amount of code required, probably.

This message was sent by Atlassian JIRA

View raw message