hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HCFS/Progress" by JayVyas
Date Fri, 04 Apr 2014 11:31:53 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HCFS/Progress" page has been changed by JayVyas:
https://wiki.apache.org/hadoop/HCFS/Progress?action=diff&rev1=16&rev2=17

- '''Hadoop FileSystem and FileContext Workstrea (2014) '''
+ '''Hadoop FileSystem and FileContext Workstream (2014) '''
  
- Building on the work of last year, we have now moved this workstream to concrete JIRAs.

+ Building on the work of last year, we have now moved this workstream to concrete JIRAs.
  
+ In order to begin iterating on improving HCFS test coverage we need an unambiguous mechanism
for implementing HCFS tests.  After implementing HCFS tests, we want to compare test coverage
with a gold-standard, and finally, where any gaps exist we want to be able to justify them
using simplest semantics possible, ideally in code.  The below three JIRAs address these 3
issues.  Once they are completed, HCFS testing will be much simpler.
+ 
- * HADOOP-9361: This will pave the way for defining differences between FS Implementations
that can be used to provide necessary context to FS audits in the future.
+ * HADOOP-9361: A way to define FS Implementations differences/similarities that can be used
to provide necessary context to FS audits in the future.
+ 
  * HADOOP-10463: Boosting RawLocalFileSystem test coverage and
+ 
  * HADOOP-10461: Creating an injection framework for any HCFS to plug into, which supports:
+ 
   * Ignoring or modifying expectaions of certain tests (i.e. alongside the semantics of HADOOP-9361).
+ 
   * The ceremony of inheriting or invoking necessary underlying test utility classes as necessary.
  
  In another thread, we will work to improve coverage of RawLocalFileSystem (LocalFs/LocalFileSystem)
  
  '''Hadoop FileSystem Validation Workstream (2013)'''
  
- Hadoop has a pluggable FileSystem Architecture. 3rd party FileSystems can be enabled for
Hadoop by developing a plugin that mediates between the Hadoop FileSystem Interface and the
interface of the 3rd Party FileSystem. For those developing a Hadoop FileSystem plugin, there
is no comprehensive test library to validate that their plugin creates a Hadoop FileSystem
implementation that is Hadoop compatible.  
+ Hadoop has a pluggable FileSystem Architecture. 3rd party FileSystems can be enabled for
Hadoop by developing a plugin that mediates between the Hadoop FileSystem Interface and the
interface of the 3rd Party FileSystem. For those developing a Hadoop FileSystem plugin, there
is no comprehensive test library to validate that their plugin creates a Hadoop FileSystem
implementation that is Hadoop compatible.
  
  What do we mean by comprehensive? We mean that there is a test for every single operation
in the FS Interface that properly tests the expected behavior of the operation given the full
variability of its parameters. To create a comprehensive test library, we plan to do the following:
  
  * Focus on the Hadoop 2.0 FS Interface. If possible, create a work stream that would allow
testing and validation of the FS 1.0 Interface also.
  
- * Undertake an audit of the Hadoop FileSystem 1.0 Test Coverage - [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests
| Link to Work In Progress ]]
+ * Undertake an audit of the Hadoop FileSystem 1.0 Test Coverage - [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests|Link
to Work In Progress]]
  
- * Undertake an audit of the Hadoop FileSystem 2.0 Test Coverage - [[https://wiki.apache.org/hadoop/HCFS/FileSystem-2.0-Tests
| Link to Work In Progress ]]
+ * Undertake an audit of the Hadoop FileSystem 2.0 Test Coverage - [[https://wiki.apache.org/hadoop/HCFS/FileSystem-2.0-Tests|Link
to Work In Progress]]
  
-     - This includes an audit of the new Hadoop FS Tests added by Steve Loughran for his
[[https://issues.apache.org/jira/browse/HADOOP-8545 | Hadoop FS Plugin for SWIFT]]
+  . - This includes an audit of the new Hadoop FS Tests added by Steve Loughran for his [[https://issues.apache.org/jira/browse/HADOOP-8545|Hadoop
FS Plugin for SWIFT]]
  
  * Document the FileSystem 2.0 Specification (as a JavaDoc) as a JIRA Ticket
+ 
-     - This includes resolving and documenting the expected behavior of the FileSystem 2.0
Operations and all the FS operations - [[ https://issues.apache.org/jira/browse/HADOOP-9371
| Steve Loughran has started this already]]
+  . - This includes resolving and documenting the expected behavior of the FileSystem 2.0
Operations and all the FS operations - [[https://issues.apache.org/jira/browse/HADOOP-9371|Steve
Loughran has started this already]]
  
  * Create a gap analysis contrasting the FileSystem 2.0 Specification and the audits of existing
FileSystem 2.0 Test Coverage.
  
  * Create tests to fill in the gaps
-       
+ 
-     - Also, create a test strategy for handling Object/Block Stores as Hadoop FileSystems
+  . - Also, create a test strategy for handling Object/Block Stores as Hadoop FileSystems
  
  Once the comprehensive test library is complete, it can then be used by the provider of
a 3rd Party FileSystem to verify compatibility with Hadoop by:
-      
-     - Passing Functional Validation: Successfully passing the test library that will be
created (described above) 
  
+  . - Passing Functional Validation: Successfully passing the test library that will be created
(described above)
-     - Passing Ecosystem Validation: Successfully passing the Hadoop Integration Tests from
Apache BigTop
+  - Passing Ecosystem Validation: Successfully passing the Hadoop Integration Tests from
Apache BigTop
- 
  
  ----
- Next Meeting  
+ Next Meeting
  
  '''June 25th 2013''' - Face to Face meeting at Red Hat in Mountain View. The day before
Hadoop Summit. Details/Sign up here - http://hadoop-fs.eventbrite.com/
  
  ----
- ''Work thus far'' 
+ ''Work thus far''
  
  ----
  '''June 10th 2013''' 9AM PST via Google Hangout
@@ -66, +71 @@

  Outcome:
  
  - Validation of the current goals, plus the addition of:
-   
-   * Leveraging BigTop in order to make use of the Hadoop Ecosystem Tests as additional Test
Coverage for a FileSystem Implementation
-   
-   * Create a workstream to identify if Object/Blob stores have unique properties that make
them a special case for Test Coverage as a Hadoop FS. Create a strategy for handling Object/Block
Stores.
  
-   * Focus tests against the AbstractFileSystem class rather than the FileSystem Class (which
is an abstract class). Yes, this can be confusing.
+  * Leveraging BigTop in order to make use of the Hadoop Ecosystem Tests as additional Test
Coverage for a FileSystem Implementation
  
+  * Create a workstream to identify if Object/Blob stores have unique properties that make
them a special case for Test Coverage as a Hadoop FS. Create a strategy for handling Object/Block
Stores.
+ 
+  * Focus tests against the AbstractFileSystem class rather than the FileSystem Class (which
is an abstract class). Yes, this can be confusing.
+ 
-   * Create a Hadoop 2.0 FileSystem Interface Specification for developers creating plugins
as well as additional background for interested users. This should be created as a JavaDoc
and managed in JIRA so that it supports proper governance.
+  * Create a Hadoop 2.0 FileSystem Interface Specification for developers creating plugins
as well as additional background for interested users. This should be created as a JavaDoc
and managed in JIRA so that it supports proper governance.
  
  The workstream definition at the top of this page has been updated to reflect the new additions
to the initiative.
  
  ----
  '''June 4th 2013'''
  
- Created a [[https://github.com/wattsteve/HCFS/blob/master/jdiff/Report.txt | diff report]]
contrasting Hadoop FileSystem 1.0 and 2.0
+ Created a [[https://github.com/wattsteve/HCFS/blob/master/jdiff/Report.txt|diff report]]
contrasting Hadoop FileSystem 1.0 and 2.0
  
- Next step is to evaluate how comprehensive the unit test case coverage is for FileSystem
1.0 and 2.0. This is a work in progress [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests
| Audit of the FileSystem 1.0 Test Library ]]
+ Next step is to evaluate how comprehensive the unit test case coverage is for FileSystem
1.0 and 2.0. This is a work in progress [[https://wiki.apache.org/hadoop/HCFS/FileSystem-1.0-Tests|Audit
of the FileSystem 1.0 Test Library]]
  
  ----
  '''May 23rd 2013''' - A broader call for participation was made to the hadoop-core dev proposing:
  
- * broader participation in [[ https://issues.apache.org/jira/browse/HADOOP-9371 | defining
the expected behavior of Hadoop FileSystem operations]]
+ * broader participation in [[https://issues.apache.org/jira/browse/HADOOP-9371|defining
the expected behavior of Hadoop FileSystem operations]]
  
  * creating a comprehensive test suite verifying compliance with the expected behavior of
a Hadoop FileSystem
  
  * several google hangouts and a workshop to discuss the topics
  
+ The following parties responded that they were interested in participation: - mbhandarkar@gopivotal.com
, shv.hadoop@gmail.com , stevel@hortonworks.com , erlv5241@gmail.com , shaposhnik@gmail.com
, apurtell@apache.org , cdouglas@apache.org , jayhawk@cs.ucsc.edu , sanjay@hortonworks.com
, dmitry.bimatov@isilon.com
- The following parties responded that they were interested in participation:
- - mbhandarkar@gopivotal.com, shv.hadoop@gmail.com, stevel@hortonworks.com, erlv5241@gmail.com,
shaposhnik@gmail.com, apurtell@apache.org, cdouglas@apache.org, jayhawk@cs.ucsc.edu, sanjay@hortonworks.com,
dmitry.bimatov@isilon.com
  

Mime
View raw message