Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Eyt2FAJN6To40Urv7VljNPPWsaaDCheXnKy5XUmFMecI0G4M6Lueq8ryKMvmB9gvY6sf+T8Nx4kinHnSsOOtgMV/feDakM1v5smj7+lol5A5dISQDkrkQktU8VCkkjyRAkK02dP65W+BO7m2aPQZtmbPuMoSGhDc7r2AMSgqwtc=;
References: <E86C5DD2-6436-4D23-8479-3DD0F9B00E81@magnetic.com>
 <1362195777.67783.YahooMailNeo@web140603.mail.bf1.yahoo.com>
 <DF3A2D23-4F15-460D-9CE3-831E3AD1F7AF@magnetic.com>
 <1362245928.60994.YahooMailNeo@web140603.mail.bf1.yahoo.com>
 <F3F1DB40-AB84-44C7-AA4A-DC2BAC8A5582@magnetic.com>
Message-ID: <1362257792.73980.YahooMailNeo@web140604.mail.bf1.yahoo.com>
Date: Sat, 2 Mar 2013 12:56:32 -0800 (PST)
From: lars hofhansl <larsh@apache.org>
Reply-To: lars hofhansl <larsh@apache.org>
Subject: Re: HBase Thrift inserts bottlenecked somewhere -- but where?
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: <F3F1DB40-AB84-44C7-AA4A-DC2BAC8A5582@magnetic.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1023258982-1545744671-1362257792=:73980"

--1023258982-1545744671-1362257792=:73980
Content-Type: text/plain; charset=us-ascii

They are flushed to 3 nodes (but not sync'ed to disk on those replicas), so you'll eat 3 network RTTs.

I wrote a bit about this here: http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-durable-sync.html

You can switch a column family to deferred log flush. In that case the edit is flushed to the 3 replication asynchronously with 1 or 2 secs.
(And if even get to finish HBASE-7801, one can control this per mutation).


-- Lars


________________________________
 From: Dan Crosta <dan@magnetic.com>
To: "user@hbase.apache.org" <user@hbase.apache.org> 
Sent: Saturday, March 2, 2013 10:47 AM
Subject: Re: HBase Thrift inserts bottlenecked somewhere -- but where?
 
On Mar 2, 2013, at 12:38 PM, lars hofhansl wrote:
> "That's only true from the HDFS perspective, right? Any given region is 
> "owned" by 1 of the 6 regionservers at any given time, and writes are 
> buffered to memory before being persisted to HDFS, right?"
> 
> Only if you disabled the WAL, otherwise each change is written to the WAL first, and then committed to the memstore.
> So in the sense it's even worse. Each edit is written twice to the FS, replicated 3 times, and all that only 6 data nodes.

Are these writes synchronized somehow? Could there be a locking problem somewhere that wouldn't show up as utilization of disk or cpu?

What is the upshot of disabling WAL -- I assume it means that if a RegionServer crashes, you lose any writes that it has in memory but not committed to HFiles?


> 20k writes does seem a bit low.

I adjusted dfs.datanode.handler.count from 3 to 10 and now we're up to about 22-23k writes per second, but still no apparent contention for any of the basic system resources.

Any other suggestions on things to try?

Thanks,
- Dan
--1023258982-1545744671-1362257792=:73980--