hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joydeep Sarma <jsensa...@gmail.com>
Subject Re: commit semantics
Date Wed, 13 Jan 2010 19:01:51 GMT
i posted on the jira as well - but we should be able to simulate the
effect of the patch.

if the sync was simulated merely a sleep (for 2-3ms - whatever is the
average RTT for dfs write pipeline) instead of an actual call into dfs
client - it should simulate the effect of the patch. (the appends
would proceed in parallel, each sync would block for sometime).

so we should be able to test whether this gets a performance win for
the queue threshold=1 case.

On Wed, Jan 13, 2010 at 10:43 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> Awesome, I will try to post a patch soon and  will let you know as soon as I
> have the first version ready.
>
> thanks,
> dhruba
>
>
> On Wed, Jan 13, 2010 at 10:40 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> I'll be happy to benchmark, we already have code to test the
>> multi-client hitting 1 region server case.
>>   know
>> J-D
>>
>> On Wed, Jan 13, 2010 at 10:38 AM, Dhruba Borthakur <dhruba@gmail.com>
>> wrote:
>> > I will try to make a patch for it first. depending on the complexity of
>> the
>> > patch code, we can decide which release it can go in.
>> >
>> > thanks,
>> > dhruba
>> >
>> > On Wed, Jan 13, 2010 at 9:56 AM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>> >
>> >> That's great dhruba, I guess the sooner it could go in is 0.21.1?
>> >>
>> >> J-D
>> >>
>> >> On Wed, Jan 13, 2010 at 8:51 AM, Dhruba Borthakur <dhruba@gmail.com>
>> >> wrote:
>> >> > I opened http://issues.apache.org/jira/browse/HDFS-895 for this one.
>> >> >
>> >> > thanks,
>> >> > dhruba
>> >> >
>> >> > On Tue, Jan 12, 2010 at 9:41 PM, Joydeep Sarma <jsensarma@gmail.com>
>> >> wrote:
>> >> >
>> >> >> this is internal to the dfsclient. this would explain why performance
>> >> >> would suck with queue threshold of 1.
>> >> >>
>> >> >> leave it up to Dhruba to explain the details.
>> >> >>
>> >> >> On Tue, Jan 12, 2010 at 9:16 PM, stack <stack@duboce.net>
wrote:
>> >> >> > On Tue, Jan 12, 2010 at 9:12 PM, stack <stack@duboce.net>
wrote:
>> >> >> >
>> >> >> >> > any IO to a HDFS-file (appends, writes, etc) ae actually
blocked
>> on
>> >> a
>> >> >> >> > pending sync. "sync" in HDFS is a pretty heavyweight
operation
>> as
>> >> it
>> >> >> >> stands.
>> >> >> >>
>> >> >> >> i think this is likely to explain limited throughput with
the
>> default
>> >> >> >> write queue threshold of 1. if the appends cannot make
progress
>> while
>> >> >> >> one is waiting for the sync - then the write pipeline
is going to
>> be
>> >> >> >> idle most of the time (with queue threshold of 1).
>> >> >> >>
>> >> >> >> i think it would be good to have the sync not block other
writers
>> on
>> >> >> >> the file/pipeline. logically - it's not clear why it needs
to
>> (since
>> >> >> >> the sync is just a wait for the completion as of some
write
>> >> >> >> transaction id - allowing new ones to be queued up subsequently).
>> >> >> >
>> >> >> >
>> >> >> > Are you talking about internal to DFSClient Joydeep?  Or
some
>> >> >> > synchronization block up in hlog?
>> >> >> >
>> >> >> > St.Ack
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Connect to me at http://www.facebook.com/dhruba
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Connect to me at http://www.facebook.com/dhruba
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

Mime
View raw message