hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mukund murrali <mukundmurra...@gmail.com>
Subject New Region Server failing to open region and subsequent cluster hang
Date Thu, 20 Aug 2015 11:58:06 GMT
Hi

Recently we tried adding a new regionserver to the cluster. The region
server had the following logs for all the regions it tried to open.

2015-08-14 01:23:52,175 ERROR [RS_OPEN_REGION-localhost:16020-0]
handler.OpenRegionHandler: Failed open of
region=testnamespace:testtable,test_14390754745,1439165529250.7aaa895f2ec3ca602c50fcc69120c01e.,
starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
        at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4566)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4873)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801)
        at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752)
        at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)
        at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)
        at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4563)
        ... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
        at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:601)
        at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:578)
        ... 15 more


In addition to this, all other region server began to stop serving request.

One thing to note is during this time a major compaction was running for a
table.

Also the hbase:meta table was in the new rs.

The clients had the exception

org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location


and jstack of clients were waiting at locateRegionInMeta.

On decommissioning the new node, the existing regionserver started serving
request. What would have caused this problem?  We are using hbase-1.0.0.

Thanks

Regards
Mukund Murrali

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message