hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sagar naik <sn...@attributor.com>
Subject Regions failed to migrate
Date Mon, 05 Dec 2011 18:45:02 GMT
Scenario:
  -  regionserver machine was rebooted . AWS random reboot .
Regionserver logs show shutdown

  - Datanode which is on same machine also recvd a kill command

  - Master did not migrate  the regions. It did detect the node down

  - I checked after 6 hours , the hbase was in inconsistent state . (hbck)

 - When I restarted the regionserver it was giving me a UnknownScannerException

 - I ended up, restart master only and then regionserver would start fine

 - After about 1 hr of regionserver going down, major compaction
(croned) kicked in

Question:

    Why did regions on the regionserver did not migrate ? am I missing
something, some config params.
    Most of the config is default except for compaction interval

Thanks


REGIONSERVER LOGS:

2011-12-04 22:19:09,586 INFO
org.apache.hadoop.hbase.regionserver.wal.HLog: moving old hlog file
/user/nileus/hbase-storage/.logs/ip-X-X-X-X,60020,1322010465849/ip-10-174-43-151.us-west-1.compute.internal%3A60020.1322967102356
whose highest sequenceid is 127636589 to
/user/nileus/hbase-storage/.oldlogs/ip-X-X-x-X%3A60020.1322967102356
2011-12-04 22:43:47,291 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting; hbase.shutdown.hook=true;
fsShutdownHook=Thread[Thread-15,5,main]
2011-12-04 22:43:47,291 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
hook


MASTER LOG:
2011-12-04 22:46:39,133 INFO
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
ephemeral node deleted, processing expiration
[datanode001,60020,1322010465849]
2011-12-04 22:46:39,133 INFO
org.apache.hadoop.hbase.zookeeper.RegionServerTracker: No HServerInfo
found for datanode001,60020,1322010465849
2011-12-04 22:47:20,828 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
2011-12-04 22:52:20,893 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
2011-12-04 22:57:20,959 INFO
org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
servers=3 regions=421 average=140.33333 mostloaded=141 leastloaded=141
2


POST RESTART REGIONSERVER LOGS

2011-12-05 07:21:04,589 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1809)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
2011-12-05 07:21:14,469 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1809)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Mon Dec  5 07:21:25 PST 2011 Killing regionserver
2011-12-05 07:21:25,647 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting; hbase.shutdown.hook=true;
fsShutdownHook=Thread[Thread-15,5,main]
2011-12-05 07:21:25,647 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
hook

Mime
View raw message