hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Weiner <...@gumgum.com>
Subject Re: Client with HTablePool can't connect after server restart
Date Thu, 22 Dec 2011 19:01:31 GMT
Just an update on this thread:

JD told me via IRC that this problem happens because my HBase manages
Zookeeper and when I restart HBase, Zookeeper is restarted along with it
and the Zookeeper client has a bug that doesn't allow it to work after a
Zookeeper restart.

He suggested a workaround which I have tested and it works:

Set HBase to no longer manage Zookeeper in hbase-env.sh:
    export HBASE_MANAGES_ZK=false
Start Zookeeper separately before starting HBase:
    hbase-daemon.sh start zookeeper

After doing that, I can stop and start HBase with stop-hbase.sh and
start-hbase.sh.  Zookeeper remains up and I no longer have to restart my
Tomcat web servers which have HTablePool clients.

I'm not sure if the Zookeeper bug has been filed or fixed yet.  If not, I
hope to get some time to investigate deeper.


On Wed, Dec 21, 2011 at 3:29 PM, Ken Weiner <ken@gumgum.com> wrote:

> I experienced the problem in production with a fully distributed setup.  I
> then reproduced the same problem in local mode from which I copied the
> error messages.  The messages in production were exactly the same except
> that they contained a server name rather than "localhost".
> Should I expect to be able to restart HBase without having to restart my
> clients (each web server in a cluster has its own HTablePool)?
> On Wed, Dec 21, 2011 at 3:20 PM, lars hofhansl <lhofhansl@yahoo.com>wrote:
>> Is this is local mode? In local mode ports of Master and RegionServer
>> might change and the client cannot (be expected to) deal with that.
>> ________________________________
>>  From: Ken Weiner <ken@gumgum.com>
>> To: hbase-user@hadoop.apache.org
>> Sent: Wednesday, December 21, 2011 3:06 PM
>> Subject: Client with HTablePool can't connect after server restart
>> Our Java client uses HTablePool and is not able to connect to the
>> server after a server reboot (stop-hbase.sh followed by start-hbase.sh).
>> Partial stack trace:
>> java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
>> at
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
>>          ...
>> at
>> org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:133)
>> at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:96)
>> Furthermore, the following message is continuously printed:
>> - Socket connection established to localhost/, initiating
>> session
>> - Unable to read additional data from server sessionid 0x1345e2d625d0009,
>> likely server has closed socket, closing socket connection and attempting
>> reconnect
>> After an HBase restart, is there any way to restore the HBase client
>> (which
>> is part of a web application) without having to restart it?
>> Environment:
>> - HBase 0.90.4
>> - HBase-Managed Zookeeper 3.3.2-1031432
>> - Ubuntu Server
>> Thanks,
>> -Ken

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message