hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: commit semantics
Date Wed, 13 Jan 2010 05:12:23 GMT
(Below is a note from Joydeep.  Something about Joydeeps' messages are
requiring that I approve/disapprove them.  For the message below, I his
disapprove by mistake so am copying it here manually)

---------- Forwarded message ----------
From: Joydeep Sarma <jssarma@apache.org>
To: hbase-dev@hadoop.apache.org, kannan@facebook.com, Dhruba Borthakur <
dhruba@facebook.com>
Date: Tue, 12 Jan 2010 15:39:05 -0800
Subject: Re: commit semantics
btw - i followed up with Dhruba afterwards on this comment:

> any IO to a HDFS-file (appends, writes, etc) ae actually blocked on a
> pending sync. "sync" in HDFS is a pretty heavyweight operation as it
stands.

i think this is likely to explain limited throughput with the default
write queue threshold of 1. if the appends cannot make progress while
one is waiting for the sync - then the write pipeline is going to be
idle most of the time (with queue threshold of 1).

i think it would be good to have the sync not block other writers on
the file/pipeline. logically - it's not clear why it needs to (since
the sync is just a wait for the completion as of some write
transaction id - allowing new ones to be queued up subsequently).

Joydeep

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message