hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: HDFS-895 hflush speedup
Date Thu, 15 Apr 2010 19:31:10 GMT
I'm running whats on here:

http://github.com/ryanobjc/hadoop-cloudera

Which is basically the patches you sent me previously.

I have a cluster I would be willing to test with...

-ryan

On Thu, Apr 15, 2010 at 11:54 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Apr 15, 2010 at 10:51 AM, Kannan Muthukkaruppan <Kannan@facebook.com
>> wrote:
>
>> Todd: Good stuff! Have you tried this out on a HBase test too?
>>
>>
> Not yet - I basically had a burst of inspiration last night around 11pm,
> coded til I couldn't stay awake, and threw up a patch :) I hope to give it a
> try today (and also update patch to trunk so I can get some reviews there)
>
>
>> > I think with it on, we can practically turn off group commit and get
>> quite a bit speedup.
>>
>> Will batching now happen underneath the DFSClient automatically now that
>> syncs/appends are not serialized.
>>
>>
> It's not quite batching, but essentially I think we get the same benefit.
> While one thread is waiting for *its* packet/write to be synced, another
> thread can still append to the stream and call sync for its own data.
>
> -Todd
>
> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, April 15, 2010 1:59 AM
>> To: hbase-dev@hadoop.apache.org
>> Subject: HDFS-895 hflush speedup
>>
>> Any of you out there who are running the 0.20+200+etc hadoop with 0.20
>> branch hbase want to try this patch? I think with it on, we can practically
>> turn off group commit and get quite a bit speedup.
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message