hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brent Miller <brentalanmil...@gmail.com>
Subject Client still attempting to connect to failed regionserver
Date Fri, 05 Aug 2011 21:13:58 GMT
I've been evaluating HBase for an upcoming project, and must say I'm quite
impressed with the preference.

I've been using a test client to simulate the load that we're expecting.
This morning we had one of the regionservers die and we're finding that the
test application is still trying to reconnect to the failed regionserver,
even after restarting the application. (hadoop-3 is the failed server)

When the client starts up, we see the following exception:

11/08/05 13:21:34 ERROR [GENTEST7] test.Main$DataGen: Caught exception while
inserting data
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
5578 actions: servers with issues: hadoop-3.ionamerica.priv:60020,
 at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1241)
 at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:826)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:682)
 at org.apache.hadoop.hbase.client.HTable.put(HTable.java:667)
at test.Main$DataGen.run(Main.java:196)
 at java.lang.Thread.run(Thread.java:679)

And the master's log is *filled* with:

2011-08-05 13:30:56,349 INFO
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Received
exception accessing META during server shutdown of
hadoop-3.ionamerica.priv,60020,1312306642172, retrying META read

I was under the assumption that if a regionserver failed, the clients would
automatically switch over to a good regionserver. Also, if I pull up the
mater's web UI, it no longer shows the failed regionserver in the "Region
Servers" section. Is this a bug or does the client have to somehow check if
a regionserver is valid?

We're using Clouder'a HBase 0.90.3-cdh3u1 on Ubuntu 10.04

This seems similar to
http://mail-archives.apache.org/mod_mbox/hbase-user/201106.mbox/%3C7B3A9A088A1B88488CBD26C63C1581D40377C7C3@ex-01%3E
but
there doesn't seem to be any resolution there.

Thanks,
Brent

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message