hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HCFS/Progress" by JayVyas
Date Tue, 02 Sep 2014 11:28:55 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HCFS/Progress" page has been changed by JayVyas:
https://wiki.apache.org/hadoop/HCFS/Progress?action=diff&rev1=20&rev2=21

+ '''How to update the HCFS contract and test classes as the FileSystem evolves.'''
+ 
+ When we define new file system behaviours, its critical to update the contract documentation
and tests.  However, its not particularly difficult to do this,
+ 
+ the HCFS contracts only consist of:
+ 
+  1. Unit tests which are extended for file systems, annotated with fields from
+  1. XML files which define a filesystem's semantics
+  1. A series of .md files, which define a semi formal specification.
+ 
+ The steps to extend any FileSystem semantics are now quite simple, and explicit.
+ 
+ 1) Check out the guide [[https://github.com/apache/hadoop-common/blob/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/extending.md|here]],
and update the .md contract files.
+ 
+ 2) Update the existing unit tests (test/java/org/apache/hadoop/fs/contract/) where relevant.
+ 
  '''Hadoop FileSystem and FileContext work, largely concluded.'''
  
- Good news ! With https://issues.apache.org/jira/browse/HADOOP-9361, we are now able to test
hadoop FileSystems in an unambiguous and declarative manner,
+ Good news ! With https://issues.apache.org/jira/browse/HADOOP-9361, we are now able to test
hadoop FileSystems in an unambiguous and declarative manner, using a combination of:
- using a combination of:
  
  * An XML File to define FileSystem semantics.  This file needs to be loaded in your unit
tests.  The contract will define the semantics of your file system, and the unit tests will
then test based on the parameters you define.  For example,
+ 
  {{{
     <property>
     <name>fs.contract.supports-unix-permissions</name>
@@ -15, +31 @@

  * The standard contract test super classes bundled into Hadoop.  These are built in the
hadoop common tests jar (hadoop-common-3.0.0-SNAPSHOT-tests.jar).
  
  * Adding custom classes to override each of the above super classes.  To do this, you manually
create classes extending from the super classes in the hadoop tests jar, like so:
+ 
  {{{
   public class TestMyHCFSBaseContract extends AbstractFSContract
   public class TestMyHCFSCreateTests extends AbstractContractCreateTest
@@ -25, +42 @@

  }}}
  And so on (all the classes which you can overide are in org.apache.hadoop.fs.contract.,
and you can scan the existing hadoop source code for examples of how to properly override
them.
  
+ The completion of this coherent and flexible test framework allows us to expand upon and
customize hadoop file system work.  To extend the contract tests, or add new semantics, there
is a clear path : The .md files, which exist inside of existing hadoop-common source code.
 See the src/site/markdown/filesystem/.... files to do so. These can easily be browsed here:
- The completion of this coherent and flexible test framework allows us to expand upon and
customize hadoop file system work.  To extend the contract tests, or add new semantics,
- there is a clear path : The .md files, which exist inside of existing hadoop-common source
code.  See the src/site/markdown/filesystem/.... files to do so.
- These can easily be browsed here:
  
  https://github.com/apache/hadoop-common/tree/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem
  
@@ -50, +65 @@

  * BIGTOP-1089: Scale testing as a universal HCFS integration test, confirming that the whole
ecosystem works together on FS interface level.  (Scale testing / updateing to 50 input splits
pending from OrangeFS community).
  
  In another thread, we will work to improve coverage of RawLocalFileSystem (LocalFs/LocalFileSystem)
- 
  
  '''Hadoop FileSystem Validation Workstream (2013)'''
  

Mime
View raw message