hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: commit semantics
Date Wed, 13 Jan 2010 16:51:13 GMT
I opened http://issues.apache.org/jira/browse/HDFS-895 for this one.

thanks,
dhruba

On Tue, Jan 12, 2010 at 9:41 PM, Joydeep Sarma <jsensarma@gmail.com> wrote:

> this is internal to the dfsclient. this would explain why performance
> would suck with queue threshold of 1.
>
> leave it up to Dhruba to explain the details.
>
> On Tue, Jan 12, 2010 at 9:16 PM, stack <stack@duboce.net> wrote:
> > On Tue, Jan 12, 2010 at 9:12 PM, stack <stack@duboce.net> wrote:
> >
> >> > any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a
> >> > pending sync. "sync" in HDFS is a pretty heavyweight operation as it
> >> stands.
> >>
> >> i think this is likely to explain limited throughput with the default
> >> write queue threshold of 1. if the appends cannot make progress while
> >> one is waiting for the sync - then the write pipeline is going to be
> >> idle most of the time (with queue threshold of 1).
> >>
> >> i think it would be good to have the sync not block other writers on
> >> the file/pipeline. logically - it's not clear why it needs to (since
> >> the sync is just a wait for the completion as of some write
> >> transaction id - allowing new ones to be queued up subsequently).
> >
> >
> > Are you talking about internal to DFSClient Joydeep?  Or some
> > synchronization block up in hlog?
> >
> > St.Ack
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message