hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: commit semantics
Date Wed, 13 Jan 2010 17:56:09 GMT
That's great dhruba, I guess the sooner it could go in is 0.21.1?

J-D

On Wed, Jan 13, 2010 at 8:51 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> I opened http://issues.apache.org/jira/browse/HDFS-895 for this one.
>
> thanks,
> dhruba
>
> On Tue, Jan 12, 2010 at 9:41 PM, Joydeep Sarma <jsensarma@gmail.com> wrote:
>
>> this is internal to the dfsclient. this would explain why performance
>> would suck with queue threshold of 1.
>>
>> leave it up to Dhruba to explain the details.
>>
>> On Tue, Jan 12, 2010 at 9:16 PM, stack <stack@duboce.net> wrote:
>> > On Tue, Jan 12, 2010 at 9:12 PM, stack <stack@duboce.net> wrote:
>> >
>> >> > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on
a
>> >> > pending sync. "sync" in HDFS is a pretty heavyweight operation as it
>> >> stands.
>> >>
>> >> i think this is likely to explain limited throughput with the default
>> >> write queue threshold of 1. if the appends cannot make progress while
>> >> one is waiting for the sync - then the write pipeline is going to be
>> >> idle most of the time (with queue threshold of 1).
>> >>
>> >> i think it would be good to have the sync not block other writers on
>> >> the file/pipeline. logically - it's not clear why it needs to (since
>> >> the sync is just a wait for the completion as of some write
>> >> transaction id - allowing new ones to be queued up subsequently).
>> >
>> >
>> > Are you talking about internal to DFSClient Joydeep?  Or some
>> > synchronization block up in hlog?
>> >
>> > St.Ack
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

Mime
View raw message