hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Good VLDB paper on WALs
Date Mon, 27 Dec 2010 19:48:31 GMT
Hi Todd,

Good paper, it would be nice to get the Flush-Pipelining technique
(described in the paper) implemented in HBase and HDFS write-ahead logs. (I
am CC-ing this to hdfs-dev@hadoop as well)

HDFS currently uses Hadoop RPC and the server thread blocks till the WAL is
written to disk. In earlier deployments, I thought we could safely ignore
flush-pipelining by creating more server threads. But in our largest HDFS
systems, I am starting to see  20% sys-time usage on the namenode machine;
most of this  could be thread scheduling. If so, then it makes sense to
enhance the logging code to release server threads even before the WAL is
flushed to disk (but, of course, we still have to delay the transaction
response to the client till the WAL is synced to disk).

Does anybody have any idea on how to figure out what percentage of the above
sys-time is spent in thread scheduling vs the time spent in other system
calls (especially in the Namenode context)?


On Fri, Dec 24, 2010 at 8:17 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Via Hammer - I thought this was a pretty good read, some good ideas for
> optimizations for our WAL.
> http://infoscience.epfl.ch/record/149436/files/vldb10aether.pdf
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera

Connect to me at http://www.facebook.com/dhruba

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message