hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Singh <akshay_i...@yahoo.com>
Subject Slow start of HBase operations with YCSB, possibly because of zookeeper ?
Date Tue, 15 Jan 2013 01:04:32 GMT
 
Hi hbase users,

I am running HBase (on top of HDFS) in
distributed mode (on 8 VMs), and things like JPS look fine on all the
machines in the cluster. I am also able to run hbase shell and
interact with HBase though it. But when I want to benchmark my HBase
cluster with YCSB (Yahoo! Cloud System Benchmark,
https://github.com/brianfrankcooper/YCSB/) I see this weird problem
of slow start of the HBase operations and then picking up later.

Basically when I start the YCSB
workload from a client machine, I see these problems in chronological
order :

1) ERROR zookeeper.ZooKeeperWatcher: ZK
is null on connection event

###########
ERROR zookeeper.ZooKeeperWatcher: ZK is
null on connection event -- see stack trace for the stack trace when
constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR
STACK TRACE FOR DEBUGGING
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1322)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:584)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at
org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172)
at
org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:131)
at
com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:155)
###########

2) org.apache.zookeeper.ClientCnxn -
Error while calling watcher
java.lang.NullPointerException: ZK
is null

############
ERROR org.apache.zookeeper.ClientCnxn -
Error while calling watcher
java.lang.NullPointerException: ZK is
null
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:334)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
############

3) And then finally it starts the
operation on HBase (which means Zookeeper is running fine and can be connected to )

4) The operations remains below 10
ops/sec for first 60-70 sec, and then grow gradually to reach aroun
1300 ops/sec (normally expected number)

Here are the actual logs :: http://pastebin.com/NC1zKwRF

I am running
1) Hadoop-1.0.1
2) HBase-0.94.1
3) Zookeeper-3.3.6
4) Java 1.6.0_24 (openJDK-6)
5) OS : Ubuntu-11.10
6) YCSB-0.14

What I have already tried :

1) Checked my DNS setting (just to be
sure .. using synced /etc/hosts file) .. no luck
2) Increasing
"hbase.zookeeper.watcher.sync.connected.wait" to 10000
(default:2000), this get rid of "ZK is null ****" errors,
but slow start is still the issue with no improvement.

I am clueless as to what may be the
reason behind this 'slowly picking up' behavior of my set-up.
Please advise.

Thanks,
Akshay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message