hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thilee Subramaniam <thi...@quantcast.com>
Subject Need to add fs shim to use QFS
Date Fri, 05 Oct 2012 17:27:37 GMT
We at Quantcast have released QFS 1.0 (Quantcast File System) to open
source. This is based on the KFS 0.5 (Kosmos Distributed File System),
a C++ distributed filesystem implementation. KFS plugs into Apache
Hadoop via the 'kfs' shim that is part of Hadoop codebase.

QFS has added support for permissions, and also, provides fault tolerance
through Reed-Solomon encoding as well as replication. There are also a
number of performance and stability improvements, including a rewrite of
the client library to allow parallel concurrent I/Os. Going forward, new
releases of KFS will come from QFS.

The open source release of QFS is at http://quantcast.github.com/qfs

QFS plugs into Apache Hadoop the same way KFS does. Currently, one would
apply the patches or JARs from the QFS source tree onto Apache Hadoop to
make Hadoop use QFS. The patch for Apache Hadoop 1.0.X can be found at

In order to make the integration seamless, we would like to add a 'qfs'
shim to Apache Hadoop so that the current active branches (1.0.X, 2.X.X,
0.23.X) of Apache Hadoop can use QFS.

Towards this, I've submited an ASF JIRA feature ticket (HADOOP-8885) under
hadoop-common project, and send a pull request with the QFS shim changes
to https://github.com/apache/hadoop-common/tree/branch-1.0.2

I will subsequently submit pull requests to the other active Hadoop

If you have any question, I will be happy yo answer or provide more
details on QFS.

- Thilee

View raw message