hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HCFS/FileSystem-2.0-Tests" by JayVyas
Date Fri, 21 Jun 2013 03:10:34 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HCFS/FileSystem-2.0-Tests" page has been changed by JayVyas:
https://wiki.apache.org/hadoop/HCFS/FileSystem-2.0-Tests?action=diff&rev1=5&rev2=6

  = Tests added in package org.apache.hadoop.fs , for the FileSystem API, in 2.0: =
+ There are essentially 2 major types of tests in this package which are relevant to the HCFS
working group.   1) Those that exercise the FileSystem and 2) Those that exercise the new
AbstractFileSystem.
- 
- There are essentially 2 major types of tests in this package which are relevant to the HCFS
working group.  
- 1) Those that exercise the FileSystem and
- 2) Those that exercise the new AbstractFileSystem.
  
  == FileSystem Tests ==
- 
- These tests build off of the  [[HCFS/FileSystem-1.0-Tests|FileSystem-1.0]] test library.

+ These tests build off of the  [[HCFS/FileSystem-1.0-Tests|FileSystem-1.0]] test library.
- 
  ||File System Implementation ||Class Name ||Description ||
  ||HCFS ||TestFileSystemCanonicalization ||Checks "canonicalization" of the file system URL
||
  ||MOCK ||TestFileSystemInitialization ||Check if FileSystem can be properly initialized
if URLStreamHandlerFactory is set ||
@@ -17, +12 @@

  ||ABSTRACT ||TestFilterFileSystem || ||
  ||ABSTRACT ||TestFilterFs || ||
  ||LOCAL ||TestFsShellCopy || ||
- ||HDFS  ||TestHarFileSystemBasics ||This test class checks basic operations with HarFileSystem
including various initialization cases, getters, and modification methods. ||
+ ||HDFS ||TestHarFileSystemBasics ||This test class checks basic operations with HarFileSystem
including various initialization cases, getters, and modification methods. ||
  ||LOCAL ||TestListFiles ||This class tests the FileStatus API. ||
- ||LOCAL || TestLocalFileSystemPermission || Test permissions on local fs impl ||
+ ||LOCAL ||TestLocalFileSystemPermission ||Test permissions on local fs impl ||
  ||LOCAL ||TestLocalFSFileContextCreateMkdir || ||
  ||LOCAL ||TestLocalFSFileContextMainOperations || ||
  ||LOCAL ||TestLocalFSFileContextSymlink ||Test symbolic links using FileContext and LocalFs.
||
  ||LOCAL ||TestLocalFsFCStatistics ||Tests the File Context Statistics for LocalFileSystem
||
  ||LOCAL ||TestLocal_S3FileContextURI || ||
- ||S3 ||TestS3_LocalFileContextURI ||  ||
+ ||S3 ||TestS3_LocalFileContextURI || ||
  ||Generic ||TestTrash ||Tests trash intervals ||
  
+ 
+ 
+ 
  == AbstractFileSystem Tests ==
-   
- These are new to Hadoop 2.X.  Many of these tests define a protected parameter which for
the FileContext which 
- can be overriden by extending classes, making them well suited for HCFS testing in 2.0 AbstractFileSystem
implementations. 
+ These are new to Hadoop 2.X.  Many of these tests define a protected parameter which for
the FileContext which  can be overriden by extending classes, making them well suited for
HCFS testing in 2.0 AbstractFileSystem implementations.
+ ||HCFS ||FCStatisticsBaseTest ||Base Test to FileContext ||
+ ||HCFS ||FileContextCreateMkdirBaseTest ||Base Test for mkdir operations ||
+ ||HCFS ||FileContextMainOperationsBaseTest ||Base Test for FileContext operations (similar
to TestFileSystem) ||
+ ||HCFS ||FileContextPermissionBase ||Generic test for FileContext permissions ||
+ ||HCFS ||FileContextSymlinkBaseTest ||Testing of symlinks creation and handling ||
+ ||HCFS ||FileContextURIBase ||Testing multiple filesystem URIs together ||
+ ||HCFS ||FileContextUtilBase ||Testing util() provided by FileContext ||
+ ||HCFS ||TestAfsCheckPath ||Testing handling of paths with different schemes ||
+ ||LOCAL ||TestFcLocalFsPermission ||? ||
+ ||LOCAL ||TestFcLocalFsUtil || ||
+ ||LOCAL ||TestFileContextDeleteOnExit || ||
+ ||HCFS ||TestFileContextResolveAfs || ||
+ || ||TestFilterFs || ||
  
+ 
+ 
- || HCFS || FCStatisticsBaseTest  || Base Test to FileContext ||
- || HCFS || FileContextCreateMkdirBaseTest  || Base Test for mkdir operations||
- || HCFS || FileContextMainOperationsBaseTest  || Base Test for FileContext operations (similar
to TestFileSystem) ||
- || HCFS || FileContextPermissionBase  || Generic test for FileContext permissions ||
- || HCFS || FileContextSymlinkBaseTest  || Testing of symlinks creation and handling ||
- || HCFS || FileContextURIBase  || Testing multiple filesystem URIs together ||
- || HCFS || FileContextUtilBase  || Testing util() provided by FileContext ||
- || HCFS || TestAfsCheckPath  || Testing handling of paths with different schemes ||
- || LOCAL || TestFcLocalFsPermission  || ? ||
- || LOCAL || TestFcLocalFsUtil || ||
- || LOCAL || TestFileContextDeleteOnExit || ||
- || HCFS || TestFileContextResolveAfs || ||
- || || TestFilterFs || ||
  
  == Other Tests ==
+ ||FileContextTestHelper ||
+ ||TestAvroFSInput ||
+ ||TestCommandFormat ||
+ ||TestDFVariations ||
+ ||TestDelegationTokenRenewer ||
+ ||TestFileStatus ||
+ ||TestFsOptions ||
+ ||TestFsShellCopy ||
  
- || FileContextTestHelper ||
- || TestAvroFSInput  ||
- || TestCommandFormat  ||
- || TestDFVariations  ||
- || TestDelegationTokenRenewer  ||
- || TestFileStatus  ||
- || TestFsOptions  ||
- || TestFsShellCopy  ||
  
  == Sub packages, specific implementation tests ==
+ ||kfs ||
+ ||loadGenerator ||
+ ||permission ||
+ ||s3 ||
+ ||s3native ||
+ ||shell ||
+ ||slive ||
+ ||viewfs ||
  
- || kfs ||
- || loadGenerator ||
- || permission ||
- || s3 ||
- || s3native ||
- || shell ||
- || slive ||
- || viewfs ||
+ 
+ '''Other Relevant Test Frameworks''' (Swift, KFS, etc...)
+ 
+ ''Swift-File-System ''
+ 
+ Many of these tests are decoupled from swift semantics by inheriting from the base class
"SwiftFileSystemBaseTest", and thus, they may be adoptable as HCFS tests readily.  In addition
to generic tests, the swift-file-system test suite also includes Swift specific functional
tests (for example, reconciliation of swift object paths with files ).  These are taken from
Steve Loughran's github Fork of the [[https://github.com/steveloughran/Hadoop-and-Swift-integration/|swift-file-system]]
repository.
+ ||Generic (extends SwiftFileSystemBaseTest.java) ||TestSwiftFileSystemExtendedContract ||Test
semantics of file operations in corner cases (i.e. opening a file that doesnt exist) ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||TestSwiftFileSystemRename ||Testing that renames
are consistent (i.e. after rename, file contents are in new path, etc..) .  Also tests that
invalid renames handled appropriately. ||
+ ||SwiftFS ||TestSwiftFileSystemContract ||Implementation of Swift specific hooks for the
generic hadoop FileSystemContractTest ||
+ ||SwiftFS ||TestSwiftFileSystemPartitionedUploads ||Test Swift specific partitioning behavior
on writes ||
+ ||SwiftFS ||SwiftFileSystemBaseTest ||Generic Base test class which can be failed by subclasses.
 From HCFS perspective, this class decouples subclasses from Swift semantics, that they can
mostly agnostic to FileSystem impl ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||TestSwiftFileSystemRead ||Tests behaviour
when "overreading" a file ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||TestSwiftFileSystemDelete ||Tests delete semantics
and recursive delete policy ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||TestSwiftFileSystemLsOperations ||Tests semantics
of list operations and corner cases (i.e. ls empty directorys, ls on root directory, etc..
) ||
+ ||SwiftFS ||NativeSwiftFileSystemContractBaseTest ||Implementation of Swift specific hooks
for Generic FileSYstemContractBaseTest - extended by TestSwiftFileSystemContract ||
+ ||SwiftFS ||TestSwiftObjectPath ||Tests that files represented as object paths in SwiftFS
are consistent and correct ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||TestSwiftFileSystemConcurrency ||Tests concurrent
operations using Java threads. Generic tests but extends SwiftFSTest ||
+ ||SwiftFS ||TestSwiftFileSystemBasicOps ||Hardcoded to SwiftFS in some places, but appears
generalizable - tests basic fs operations (put, create, delete, ...) ||
+ ||SwiftFS ||TestSwiftFileSystemDirectories ||Test file and directory semantics. Adaptable
to a generic test. ||
+ ||Generic (extends SwiftFileSystemBaseTest) ||SwiftFileSystemTestBlockLocation ||Tests that
blocks are provided in FileSystem getFileBlockLocations call ||
+ 
+ 
+ 
+ 
+ ''Other tests in the Hadoop-Swift-Integration repository:''
+ 
+ ||InMemorySwiftObjectStore ||An admittedly "unrealistic" mock of Swift storage engine for
testing. ||
+ ||http ||package with unit tests for Rest clients connecting to Swift. ||
+ ||snative ||package with extended version of SwiftFileSystem specific for testing large
uploads. ||
+ 
  
  == Conclusions ==
- 
  Hadoop 2.X adds several new tests and the FileContext related tests provide extension points
for other HCFS tests.
  
- This is an initial analysis.  Comparison of missing tests would be much appreciated.  
+ This is an initial analysis.  Comparison of missing tests would be much appreciated.
  
  We have used the JDiff tool to diff between 1.X (1.1) and the 2.X branches (2.1-beta) and
summarized classes by inspection in order to identify the ability to generalize these tests,
but this may be prone to error.  Comments and/or edits are welcome.
  
+ Meanwhile, other frameworks (i.e. swift) have implemented their own tests which in many
cases can be abstracted into HCFS tests which are directly applicable to any FileSystem implementation.
+ 
+ More work will need to be done to normalize and integrate the corpus of existing tests into
a single, robust, implementation independent testing suite that the community can share.
+ 

Mime
View raw message