hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Graham <billgra...@gmail.com>
Subject Re: hbase crash after restart
Date Fri, 15 Jul 2011 18:21:03 GMT
What do you see when you do this from the ZK client:

get /hbase/root-region-server

I suspect a client somewhere registered itself in ZK. Maybe fixing the IP of
the root region server in ZK will do the trick.


On Fri, Jul 15, 2011 at 10:58 AM, Jason Chuong <
jason.chuong@cbsinteractive.com> wrote:

> Hi Dave,
>
> Yes we are and on hbase version 0.90,  I've also verify that the zookeeper
> are responding via the zk shell and logs look normal.
> Just don't understand why it's trying to connect to that ip address.
>
>
> [zk: hadoop-wkr-r1:2181(CONNECTED) 1] ls /hbase
> [splitlog, unassigned, rs, root-region-server, table, shutdown]
>
>
>
>
> On Fri, Jul 15, 2011 at 9:54 AM, Buttler, David <buttler1@llnl.gov> wrote:
>
> > You really don't need 3 zookeeper nodes for a 5 node cluster. 1 is
> > sufficient.
> > Are you managing zookeeper with hbase or independently?
> >
> > Dave
> >
> >
> > -----Original Message-----
> > From: Jason Chuong [mailto:jason.chuong@cbsinteractive.com]
> > Sent: Friday, July 15, 2011 5:17 AM
> > To: user@hbase.apache.org
> > Subject: hbase crash after restart
> >
> > Hi All,
> >
> > I have a 5-node cluster setup with 3 nodes as a part of zookeeper quorum.
> > When i restart the hbase master, the server try to connect to an unknown
> > host and then crash.
> > Anyone seen this error message before or know how to resolve this thanks
> >
> > 2011-07-15 05:10:49,158 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> > connecting to server: 10.16.129.21/10.16.129.21:50712
> > 2011-07-15 05:11:10,162 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> > connecting to server: 10.16.129.21/10.16.129.21:50712
> > 2011-07-15 05:11:31,166 INFO org.apache.hadoop.ipc.HbaseRPC: Problem
> > connecting to server: 10.16.129.21/10.16.129.21:50712
> > 2011-07-15 05:11:31,170 FATAL org.apache.hadoop.hbase.master.HMaster:
> > Unhandled exception. Starting shutdown.
> > java.net.SocketTimeoutException: 20000 millis timeout while waiting for
> > channel to be ready for connect. ch :
> > java.nio.channels.SocketChannel[connection-pending remote=
> > 10.16.129.21/10.16.129.21:50712]
> > at
> >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
> > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
> > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > at $Proxy6.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
> > at
> >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.get
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message