hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Add Append-HBase support in upcoming 20.205
Date Thu, 01 Sep 2011 22:36:25 GMT

On Aug 31, 2011, at 11:41 AM, sanjay Radia wrote:

> I propose that the 20-append patches (details below)  be included in 20.205 which will
become the first official Apache
> release of Hadoop that supports Append and HBase.

> Background:
> There hasn't been a official Apache release that supports HBase. 
> The HBase community have instead been using the 20-append branch; the patches were contributed
by the HBase community including Facebook. The Cloudera distribution has also included these
> Andrew Purtell has ported these patches to 20-security branch.


I think it's high time (I thought so too last Dec too: http://s.apache.org/jr) we had an official
Hadoop release which supports HBase. Thanks for all the effort Andrew - it will be really
nice to have an Apache 0.20 with security+append!


> Risk Level:
> These patches have been used and tested on large HBase clusters by FB , by those who
use 20-append branch directly (various users including a 500 node HBase cluster at Yahoo)
and by those that use the Cloudera distribution. We have reviewed the patches and have conducted
further tests; testing and validation continues.
> Patches:
> HDFS-200. Support append and sync for hadoop 0.20 branch.
> HDFS-142. Blocks that are being written by a client are stored in the blocksBeingWritten
> HDFS-1057.  Concurrent readers hit ChecksumExceptions if following a writer to very end
of file
> HDFS-724.  Use a bidirectional heartbeat to detect stuck pipeline.
> HDFS-895. Allow hflush/sync to occur in parallel with new writes to the file.
> HDFS-1520. Lightweight NameNode operation recoverLease to trigger lease recovery.
> HDFS-1555. Disallow pipelien recovery if a file is already being lease recovered.
> HDFS-1554. New semantics for recoverLease.
> HDFS-988. Fix bug where savenameSpace can corrupt edits log.
> HDFS-826. Allow a mechanism for an application to detect that datanode(s) have died in
the write pipeline.
> HDFS-630. Client can exclude specific nodes in the write pipeline.
> HDFS-1141. completeFile does not check lease ownership.
> HDFS-1204. Lease expiration should recover single files, not entire lease holder
> HDFS-1254. Support append/sync via the default configuration.
> HDFS-1346. DFSClient receives out of order packet ack.
> HDFS-1054. remove sleep before retry for allocating a block.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message