hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Hbase stops working
Date Thu, 26 Feb 2009 12:57:42 GMT
Andrew,

For some reason, your region server shut down around 9:36:

2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.
>
> master.RegionServerOperation: process shutdown of server 127.0.0.1:40076:
> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,


But just before that the master received a startup message from it. Did you
take a look at your region server log? What does it say around that time?

Thx

J-D

On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <andrew.mccall@goroam.net>wrote:

> I've been testing a nuch on HBase and I'm running into an issue with it
> when it's been running for a while.
>
> I'm running hbase 0.19 on a single local host pseudo distributed hadoop
> 0.19 cluster.
>
> After some time I get the following in my logs:
>
> 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
> .META.,,1, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:31:55,533 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:32:18,665 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:32:35,749 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
> .META.,,1, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:33:18,771 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:33:54,889 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:33:57,387 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:34:41,579 INFO org.apache.hadoop.hbase.master.ServerManager:
> 127.0.0.1:40076 lease expired
> 2009-02-24 09:34:53,440 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
> .META.,,1, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
> .META.,,1, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:35:56,951 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:36:05,667 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP
> 2009-02-24 09:36:05,668 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:36:05,884 INFO
> org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
> server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
> numberOfMetaRegions: 1,
>  onlineMetaRegions.size(): 1
> 2009-02-24 09:36:17,689 INFO org.apache.hadoop.hbase.regionserver.HLog:
> Splitting 1 log(s) in
> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
> 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
> .META.,,1, startKey: <>, server: 127.0.0.1:40
> 076} complete
> 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:36:36,598 INFO org.apache.hadoop.hbase.regionserver.HLog: log
> file splitting completed for
> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
> 2009-02-24 09:36:56,953 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:37:05,279 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 40076, call openScanner([B@1ac5a78, [[B@6799dc, [B@1a66cfc,
> 9223372036854775807, null) from 127.0.0.1
> :50139: error: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> 2009-02-24 09:37:07,051 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
> 127.0.0.1:40076}
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>        at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>        at
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>        at
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>        at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>        at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
> 2009-02-24 09:37:07,230 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All 1 .META. region(s) scanned
> 2009-02-24 09:37:50,642 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed .META.,,1
> 2009-02-24 09:37:56,955 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
> startKey: <>, server: 127.0.0.1:40076}
> 2009-02-24 09:37:57,126 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 40076, call openScanner([B@155bf4e, [[B@82b436, [B@1056757,
> 9223372036854775807, null) from 127.0.0.1
> :35804: error: org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
> 2009-02-24 09:38:00,412 WARN org.apache.hadoop.hbase.master.BaseScanner:
> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
> 127.0.0.1:40076}
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>        at
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>        at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>        at
> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>        at
> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>        at
> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>        at
> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>        at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>
> This just repeats forever, until I restart HBase.
>
> Any help would be greatly appreciated.
> Andrew
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message