hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HCFS/Progress" by JayVyas
Date Tue, 12 Aug 2014 23:58:37 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HCFS/Progress" page has been changed by JayVyas:
https://wiki.apache.org/hadoop/HCFS/Progress?action=diff&rev1=19&rev2=20

Comment:
I beleive from now, the monolithic workstream is complete, and people can now implement and
extend / evolve the HCFS work over time, iteratively.

+ '''Hadoop FileSystem and FileContext work, largely concluded.'''
+ 
+ Good news ! With https://issues.apache.org/jira/browse/HADOOP-9361, we are now able to test
hadoop FileSystems in an unambiguous and declarative manner,
+ using a combination of:
+ 
+ * An XML File to define FileSystem semantics.  This file needs to be loaded in your unit
tests.  The contract will define the semantics of your file system, and the unit tests will
then test based on the parameters you define.  For example,
+ {{{
+    <property>
+    <name>fs.contract.supports-unix-permissions</name>
+    <value>true</value>
+    </property>
+ }}}
+ * The standard contract test super classes bundled into Hadoop.  These are built in the
hadoop common tests jar (hadoop-common-3.0.0-SNAPSHOT-tests.jar).
+ 
+ * Adding custom classes to override each of the above super classes.  To do this, you manually
create classes extending from the super classes in the hadoop tests jar, like so:
+ {{{
+  public class TestMyHCFSBaseContract extends AbstractFSContract
+  public class TestMyHCFSCreateTests extends AbstractContractCreateTest
+  public class TestMyHCFSFSContractDelete extends AbstractContractDelete
+  public class TestMyHCFSContractLoaded extends AbstractFSContractLoaded
+  public class TestMyHCFSContractMkdir extends AbstractContractMkdir
+  public class TestMyHCFSContractOpen extends AbstractContractOpenTest
+ }}}
+ And so on (all the classes which you can overide are in org.apache.hadoop.fs.contract.,
and you can scan the existing hadoop source code for examples of how to properly override
them.
+ 
+ The completion of this coherent and flexible test framework allows us to expand upon and
customize hadoop file system work.  To extend the contract tests, or add new semantics,
+ there is a clear path : The .md files, which exist inside of existing hadoop-common source
code.  See the src/site/markdown/filesystem/.... files to do so.
+ These can easily be browsed here:
+ 
+ https://github.com/apache/hadoop-common/tree/trunk/hadoop-common-project/hadoop-common/src/site/markdown/filesystem
+ 
+ Thanks to Steve Loughran and other 9361 reviewers for this critical and timely update to
the hadoop contract tests.  This democratizes storage underneath hadoop and enables continued
co-evolution of storage and YARN computation, hand in hand.
+ 
  '''Hadoop FileSystem and FileContext Workstream (2014) '''
  
  In order to begin iterating on improving HCFS test coverage we need an unambiguous mechanism
for implementing HCFS tests.  After implementing HCFS tests, we want to compare test coverage
with a gold-standard, and finally, where any gaps exist we want to be able to justify them
using simplest semantics possible, ideally in code.  The below three JIRAs address these 3
issues.  Once they are completed, HCFS testing will be much simpler.
  
- * HADOOP-9361: A programmatic way for FS implementations to broadcast the features they
support, so that all file systems can reuse the same basic test libraries without needing
to do ad-hoc overriding of certain implementations. 
+ * HADOOP-9361: A programmatic way for FS implementations to broadcast the features they
support, so that all file systems can reuse the same basic test libraries without needing
to do ad-hoc overriding of certain implementations.
  
- * HADOOP-10463: Boosting RawLocalFileSystem test coverage and 
+ * HADOOP-10463: Boosting RawLocalFileSystem test coverage and
  
  * HADOOP-10461: Creating an injection framework for any HCFS to plug into, which supports:
(feed back pending from OrangeFS Community and others).
  
@@ -17, +50 @@

  * BIGTOP-1089: Scale testing as a universal HCFS integration test, confirming that the whole
ecosystem works together on FS interface level.  (Scale testing / updateing to 50 input splits
pending from OrangeFS community).
  
  In another thread, we will work to improve coverage of RawLocalFileSystem (LocalFs/LocalFileSystem)
-  
+ 
  
  '''Hadoop FileSystem Validation Workstream (2013)'''
  

Mime
View raw message