hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Whiting <je...@qualtrics.com>
Subject Re: HBase / YCSB
Date Fri, 29 Jul 2011 17:04:31 GMT
Check the region server logs.  If they are blocking on something it should show up there. 
For cdh3 
the logs are in /var/log/hbase/.  Also you may want to turn on debug level for your logging
(either 
in log4j or in the web interface).  Finally all of your requests are going to just one region

server...npin-172-16-12-204.np.local...so it may be stuck trying to split a region or something.
 
You could try to pre-split the regions which may help.

~Jeff

On 7/29/2011 10:57 AM, Eric Hauser wrote:
> Hi,
> I've been doing different experiments with a 5-node cluster with YCSB.
>   We have been testing a number of different configurations, so I have
> been constantly been wiping our cluster up and setting it up again
> since we configure everything via Chef.   At one point, I was able to
> get the following stats from our cluster which I was pretty happy
> with:
> YCSB Client 0.1
>
> Command line: -load -db com.yahoo.ycsb.db.HBaseClient
> -Pworkloads/workloada -p columnfamily=family -p recordcount=10000000
> -s
>
> [OVERALL], RunTime(ms), 1057645.0
>
> [OVERALL], Throughput(ops/sec), 9454.96834949345
>
> [INSERT], Operations, 10000000
>
> [INSERT], AverageLatency(ms), 0.0915235
>
> [INSERT], MinLatency(ms), 0
>
> [INSERT], MaxLatency(ms), 6925
>
> [INSERT], 95thPercentileLatency(ms), 0
>
> [INSERT], 99thPercentileLatency(ms), 0
>
> [INSERT], Return=0, 10000000
>
> However, in our most recent server builds, I seem to very quickly
> deadlock something in HBase.  I've backed through all our old
> revisions and reverted a number of different configuration settings,
> and I can't seem to figure out now why the cluster is so slow.  Our
> terasort M/R tests are returning the same values as before, so I do
> not believe that there is anything wrong external to HBase.
>
> The behavior that I see when I kick off the tests is this:
>
> [UPDATE], 0, 4765
>
> [UPDATE], 1, 248
>
> [UPDATE], 2, 0
>
> [UPDATE], 3, 0
>
> [UPDATE], 4, 0
>
> Basically, it kicks off a large number of inserts and HBase grinds to
> a halt.  Some number of the writes end up getting inserted (usually
> around ~50), but then everything stops.  Here's the behavior I see
> with the region servers:
>
> npin-172-16-12-203.np.local:60030	1311956094792	requests=50,
> regions=1, usedHeap=151, maxHeap=16358
> npin-172-16-12-204.np.local:60030	1311956094776	requests=5, regions=2,
> usedHeap=157, maxHeap=16358
> npin-172-16-12-205.np.local:60030	1311956093804	requests=0, regions=0,
> usedHeap=134, maxHeap=16358
> npin-172-16-12-206.np.local:60030	1311956093809	requests=0, regions=0,
> usedHeap=134, maxHeap=16358
> npin-172-16-12-207.np.local:60030	1311956094799	requests=0, regions=0,
> usedHeap=134, maxHeap=16358
> Total:	servers: 5	 	requests=55, regions=3
>
> I did thread dumps on both the masters and region servers during this
> time and did not see anything interesting. I'm using 0.90.3-CDH3U1.
> Anyone have a suggestion on where to look next?

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com


Mime
View raw message