Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of koven2049@gmail.com
 designates 209.85.218.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTik9KXKOm6ScBRb3rT9jrQf24Ormjg@mail.gmail.com>
References: <BANLkTinhkbknJ+PjEzm3Vf96gXtL+wsiXA@mail.gmail.com>
	<B60BA63A-5159-48E3-AF5A-5F3777EB1243@cloudera.com>
	<BANLkTik9KXKOm6ScBRb3rT9jrQf24Ormjg@mail.gmail.com>
Date: Wed, 29 Jun 2011 22:39:27 +0800
Message-ID: <BANLkTi=HGhNw1sgzB4JseDUwboiGFckp3g@mail.gmail.com>
Subject: Re: performance: HLog flush to disk each thread,
 can we decrease IO calls?
From: Mingjian Deng <koven2049@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6ae1e60ba0104a6dac07b

--000e0cd6ae1e60ba0104a6dac07b
Content-Type: text/plain; charset=ISO-8859-1

Yes, it can only increase tps and less helpful for latency.
I modified my code and generated from trunk.
https://issues.apache.org/jira/browse/HBASE-4044

2011/6/29 Dhruba Borthakur <dhruba@gmail.com>

> We have implemented this idea, and it definitely increases HLog performance
> by quite a large number. The one drawabck is that writes to HLog (from HDFS
> perspective) become more "batchy", and writes to a HDFS file consume quite
> a
> bit of CPU. So I have observed that this change increase overall system
> throughput, but suffer slightly on individual transaction latency.
>
> -dhruba
>
> On Wed, Jun 29, 2011 at 6:08 AM, Joey Echeverria <joey@cloudera.com>
> wrote:
>
> > Hey Mingjian,
> >
> > This sounds like a good idea  Your patch didn't make it through. Would
> you
> > mind either filing a JIRA and uploading your patch there or at least
> posting
> > it to something like pastebin so we can take a look.
> >
> > -Joey
> >
> >
> >
> > On Jun 29, 2011, at 3:27, Mingjian Deng <koven2049@gmail.com> wrote:
> >
> > > Hi:
> > >     We found that the hlog sync to disk each time. When one thread exec
> > "doWrite(info, logKey, edit);", the others wait for "updateLock" in
> > HLog.java.
> > >     Why not the others add their edits into a list and wait. When
> sync's
> > time, the whole list sync to disk once. I think it will decrease the IO
> > calls.
> > >
> > >     So Maybe we will make two lists for edits. Each thread write to the
> > "waledits" and wait for "updateLock". Each thread can copy the "waledits"
> to
> > "flushedits" and flush the "flushedits" to
> > > disk once it gets "updateLock".
> > >
> > >     In my test, it can increase the write speed of 40%.
> > >
> > >     Just see the HLog.patch.
> > >
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

--000e0cd6ae1e60ba0104a6dac07b--