Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.161.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=AOVbUrFexZoDKebhooEoECyfCHN8MZ4w18XG3H0fYcsolGFka0eWuzpE7D9Y/OWZnv
         ktW7JvG27O7gluIw+gZjTptYGvVr+bbEQO9+e4nGoVobppEOQQMfo80JDsOn6vgbA5tD
         vPiPHsRFKMdv68h5r0BpQMHz/kvTaycGDAvIw=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <AANLkTinTRp7yz7K3p9oOzAkqz2LYru-DOikOB-SrTmur@mail.gmail.com>
References: <AANLkTik9c=1qty6PL1mg1CqdEkSmEOartW4Le9f+ETs3@mail.gmail.com>
	<AANLkTinTRp7yz7K3p9oOzAkqz2LYru-DOikOB-SrTmur@mail.gmail.com>
Date: Wed, 29 Dec 2010 11:32:19 -0800
Message-ID: <AANLkTinXp2ifdCFg3bAYxg+6LckVO7BpMoK1Q2DXd_89@mail.gmail.com>
Subject: Re: Good VLDB paper on WALs
From: Stack <stack@duboce.net>
To: dev@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Nice list of things we need to do to make logging faster (with useful
citations on current state of art).  This notion of early lock release
(ELR) is worth looking into (Jon, for high rates of counter
transactions, you've been talking about aggregating counts in front of
the WAL lock... maybe an ELR and then a hold on the transaction until
confirmation of flush would be way to go?).  Regards flush-pipelining,
it would be interesting to see if there are traces of the sys-time
that Dhruba is seeing in his NN out in HBase servers.  My guess is
that its probably drowned by other context switches done in our
servers.  Definitely worth study.

St.Ack
P.S. Minimizing context switches, a system for ELR and
flush-pipelining, recasting the server to make use of one of the DI or
OSGi frameworks, moving off log4j, etc..... Is it just me or do others
feel a server rewrite coming on?


On Mon, Dec 27, 2010 at 11:48 AM, Dhruba Borthakur <dhruba@gmail.com> wrote=
:
> HDFS currently uses Hadoop RPC and the server thread blocks till the WAL =
is
> written to disk. In earlier deployments, I thought we could safely ignore
> flush-pipelining by creating more server threads. But in our largest HDFS
> systems, I am starting to see =A020% sys-time usage on the namenode machi=
ne;
> most of this =A0could be thread scheduling. If so, then it makes sense to
> enhance the logging code to release server threads even before the WAL is
> flushed to disk (but, of course, we still have to delay the transaction
> response to the client till the WAL is synced to disk).
>
> Does anybody have any idea on how to figure out what percentage of the ab=
ove
> sys-time is spent in thread scheduling vs the time spent in other system
> calls (especially in the Namenode context)?
>
> thanks,
> dhruba
>
>
> On Fri, Dec 24, 2010 at 8:17 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Via Hammer - I thought this was a pretty good read, some good ideas for
>> optimizations for our WAL.
>>
>> http://infoscience.epfl.ch/record/149436/files/vldb10aether.pdf
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>