hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "HCFS" by SteveWatt
Date Wed, 24 Apr 2013 18:58:49 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "HCFS" page has been changed by SteveWatt:
http://wiki.apache.org/hadoop/HCFS?action=diff&rev1=2&rev2=3

  '''Hadoop Compatible File Systems (HCFS)'''
  
- Hadoop Core provides a plugin architecture for FileSystems that provides the ability to
configure Hadoop to use other File Systems besides HDFS. One example of this is the Hadoop
Amazon S3 Plugin, which exposes the Amazon S3 Service as a Hadoop Compatible File System allowing
any client of Hadoop Core (such as MapReduce and HBase) to seamlessly run on top of S3.
+ Hadoop Core provides a plugin architecture that allows one to configure Hadoop to use a
particular FileSystem via a plugin created specifically for that FileSystem. Hadoop FileSystem
plugin implementations must extend the abstract org.apache.hadoop.fs.FileSystem Class which
provides a set of operations that the FileSystem must implement. This ensures that there are
a base set of Hadoop FileSystem operations that all Hadoop Compatible FileSystems implement
and thus the core underlying FileSystem can be changed without affecting Hadoop Applications
written for Hadoop Clients such as MapReduce and HBase.
  
+ The Hadoop Distributed File System (HDFS) is the most prolifically configured File System
with Hadoop Core and is enabled via the org.apache.hadoop.hdfs.DistributedFileSystem plugin.
HDFS is somewhat different to what you get with most plugins in that it provides not only
the source code and implementation for the plugin, but also for the entire FileSystem itself.
Another example of a popular FileSystem plugin is the S3 Plugin which exposes the Amazon S3
Object Store Service as a Hadoop Compatible File System.
- ''How is this done?''
- 
- Hadoop FileSystem operations are implemented via a FileSystem plugin that extends the abstract
org.apache.hadoop.fs.FileSystem Class. The Hadoop Distributed File System (HDFS) is the most
prolific of these and extends the abstract FileSystem class with the org.apache.hadoop.hdfs.DistributedFileSystem
plugin. HDFS is however somewhat different to what you get with most plugins, in that it provides
not only the source code and implementation for the plugin, but for the entire FileSystem
itself. 
  
  In some cases, the semantics of the Hadoop FileSystem operations can be ambiguous. The community
is presently attempting to [[https://issues.apache.org/jira/browse/HADOOP-9371| define the
Semantics of the Hadoop FileSystem more rigorously]] as well as adding [[https://issues.apache.org/jira/browse/HADOOP-9258|
better test coverage for Hadoop Compatible File Systems]]
  
@@ -14, +12 @@

  
  ''What are some examples? ''
  
- LocalFileSystem, S3FileSystem and KosmosFileSystem which are all available under src/core/org/apache/hadoop/fs/
+ LocalFileSystem, S3FileSystem and KosmosFileSystem are all HCFS plugins that ship with Hadoop
and are available under src/core/org/apache/hadoop/fs/
+ 
+ Additionally, the list below includes additional 3rd Party HCFS plugins to enable additional
FileSystem for Hadoop.
  
  [[http://www.datastax.com/dev/blog/cassandra-file-system-design | CassandraFS]]
  

Mime
View raw message