hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schubert zhang <zson...@gmail.com>
Subject Fwd: RegionServer failure and recovery take a long time
Date Sat, 21 Mar 2009 06:31:33 GMT
Hi Jean Daniel,

I want your help for this issue. I attach the log files, please help analyse
it. Thanks.

Test env.
  5+1 nodes cluster.  table: create 'TESTA', {NAME => 'info', VERSIONS => 1,
COMPRESSION => 'BLOCK', BLOCKCACHE => true}

All test data is generated randomly by a program.

HRegionServer Failure (2009-03-21 08:27:41,090):
  After about 8 hours running, my node-5 rangeserver failure and the
HRegionServer Shutdown  .
  It seems caused by DFSClient exceptions. (I cannot make clear what
happened on HDFS, but it seems the HDFS is ok.)

Then I start HRegionServer at this node-5. (2009-03-21 10:53:42,747):
  After the HRegionServer started, regions were reassign. I can see the
reassign on WebGUI of HBase, since some regions are now on this node.
  But following things are blocked for a long time:
  (1) The HBase client application cannot insert data for a long time (until
2009/03/21 11:11:27, its about 18 minutes).  It is RetriesExhaustedException
exception on application side (MapReduce Job).
  (2) Some regions cannot be accessed (I cannot scan/get rows in these
regions.) The exception is NotServingRegionException when getRegion.
  (3) I check the history of the region of (2) from the WebGUI. I can see in
the history, it is assigned at 11:04:15. It is so later.
The history is:
at, 21 Mar 2009 11:10:39openRegion opened on server : nd1-rack0-cloud Sat,
21 Mar 2009 11:04:15assignmentRegion assigned to server
10.24.1.12:60020Sat, 21 Mar 2009 06:48:03openRegion opened on server :
nd1-rack0-cloud Sat,
21 Mar 2009 06:47:57assignmentRegion assigned to server
10.24.1.12:60020Sat, 21 Mar 2009 06:27:25openRegion opened on server :
nd5-rack0-cloud Sat,
21 Mar 2009 06:27:21assignmentRegion assigned to server
10.24.1.20:60020Sat, 21 Mar 2009 06:26:13openRegion opened on server :
nd5-rack0-cloud Sat,
21 Mar 2009 06:24:53assignmentRegion assigned to server
10.24.1.20:60020Sat, 21 Mar 2009 06:24:28openRegion opened on server :
nd3-rack0-cloud Sat,
21 Mar 2009 06:24:13assignmentRegion assigned to server
10.24.1.16:60020Sat, 21 Mar 2009 06:19:08openRegion opened on server :
nd4-rack0-cloud Sat,
21 Mar 2009 06:19:02assignmentRegion assigned to server
10.24.1.18:60020Sat, 21 Mar 2009 05:59:39openRegion opened on server :
nd5-rack0-cloud Sat,
21 Mar 2009 05:59:36assignmentRegion assigned to server
10.24.1.20:60020Sat, 21 Mar 2009 03:50:15openRegion opened on server :
nd3-rack0-cloud Sat,
21 Mar 2009 03:50:12assignmentRegion assigned to server
10.24.1.16:60020Sat, 21 Mar 2009 03:50:08splitRegion split from:
TESTA,13576334163@2009-03-21
00:35:57.526,1237569164012<http://nd0-rack0-cloud:60010/regionhistorian.jsp?regionname=CDR,13576334163@2009-03-21%2000:35:57.526,1237569164012>


And following is exception when I scan a rowkey range.

org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 10.24.1.12:60020 for region
TESTA,13576334163@2009-03-2100:35:57.526,1237578615553, row
'13576334163@2009-03-2100:35:57.526', but failed after 5 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
TESTA,13576334163@2009-03-21 00:35:57.526,1237578615553
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(Unknown Source)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(Unknown
Source)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(Unknown
Source)

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
TESTA,13576334163@2009-03-21 00:35:57.526,1237578615553
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(Unknown Source)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(Unknown
Source)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(Unknown
Source)

I will send the log files to you email address.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message