hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew McCall <andrew.mcc...@goroam.net>
Subject Re: Hbase stops working
Date Thu, 26 Feb 2009 13:35:07 GMT
Hi J-D,

I did check the region server logs and they didn't seem to be very  
useful at all:

Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.net
ulimit -n 1024
2009-02-23 22:11:58,817 WARN  
org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a  
distinct region server because hbase.master is set to 'local' mode

Is all that is in them, once for each time I started HBase.

I saw the shutdown message, which was preceded by this:

2009-02-24 09:34:41,579 INFO  
org.apache.hadoop.hbase.master.ServerManager: 127.0.0.1:40076 lease  
expired

I can't find any evidence it's related to my map/reduce task, it's the  
only one in the log files and it's suspiciously just before the  
process shutdown message - but, I'm new to this and honestly don't  
know what to make of it.

Andrew

On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote:

> Andrew,
>
> For some reason, your region server shut down around 9:36:
>
> 2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.
>>
>> master.RegionServerOperation: process shutdown of server  
>> 127.0.0.1:40076:
>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,
>
>
> But just before that the master received a startup message from it.  
> Did you
> take a look at your region server log? What does it say around that  
> time?
>
> Thx
>
> J-D
>
> On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <andrew.mccall@goroam.net 
> >wrote:
>
>> I've been testing a nuch on HBase and I'm running into an issue  
>> with it
>> when it's been running for a while.
>>
>> I'm running hbase 0.19 on a single local host pseudo distributed  
>> hadoop
>> 0.19 cluster.
>>
>> After some time I get the following in my logs:
>>
>> 2009-02-24 09:31:18,664 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:31:18,664 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:31:24,293 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region  
>> {regionname:
>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:31:24,293 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
>> 0,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:31:27,217 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 2 row(s) of meta region  
>> {regionname:
>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:31:27,217 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:31:55,533 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region  
>> {regionname:
>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:32:18,665 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:32:35,749 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
>> 0,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:32:38,957 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 2 row(s) of meta region  
>> {regionname:
>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:32:38,957 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:33:18,771 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:33:54,889 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region  
>> {regionname:
>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:33:57,387 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,, 
>> 0,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:34:41,579 INFO  
>> org.apache.hadoop.hbase.master.ServerManager:
>> 127.0.0.1:40076 lease expired
>> 2009-02-24 09:34:53,440 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 2 row(s) of meta region  
>> {regionname:
>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:34:53,441 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:34:53,441 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:35:38,720 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 2 row(s) of meta region  
>> {regionname:
>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:35:38,720 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:35:56,951 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:36:05,667 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer:  
>> MSG_CALL_SERVER_STARTUP
>> 2009-02-24 09:36:05,668 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.rootScanner scan of 1 row(s) of meta region  
>> {regionname:
>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:36:05,884 INFO
>> org.apache.hadoop.hbase.master.RegionServerOperation: process  
>> shutdown of
>> server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
>> numberOfMetaRegions: 1,
>> onlineMetaRegions.size(): 1
>> 2009-02-24 09:36:17,689 INFO  
>> org.apache.hadoop.hbase.regionserver.HLog:
>> Splitting 1 log(s) in
>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>> 2009-02-24 09:36:25,034 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scan of 2 row(s) of meta region  
>> {regionname:
>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>> 076} complete
>> 2009-02-24 09:36:25,034 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:36:36,598 INFO  
>> org.apache.hadoop.hbase.regionserver.HLog: log
>> file splitting completed for
>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>> 2009-02-24 09:36:56,953 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:37:05,279 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed  
>> openScanner
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC  
>> Server
>> handler 4 on 40076, call openScanner([B@1ac5a78, [[B@6799dc,  
>> [B@1a66cfc,
>> 9223372036854775807, null) from 127.0.0.1
>> :50139: error:  
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>> 2009-02-24 09:37:07,051 WARN  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>> 127.0.0.1:40076}
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>>
>>       at  
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>       at
>> sun 
>> .reflect 
>> .NativeConstructorAccessorImpl 
>> .newInstance(NativeConstructorAccessorImpl.java:39)
>>       at
>> sun 
>> .reflect 
>> .DelegatingConstructorAccessorImpl 
>> .newInstance(DelegatingConstructorAccessorImpl.java:27)
>>       at java.lang.reflect.Constructor.newInstance(Constructor.java: 
>> 513)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase 
>> .RemoteExceptionHandler 
>> .decodeRemoteException(RemoteExceptionHandler.java:95)
>>       at
>> org 
>> .apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
>> 185)
>>       at
>> org 
>> .apache 
>> .hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java: 
>> 73)
>>       at
>> org 
>> .apache 
>> .hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java: 
>> 129)
>>       at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
>> 137)
>>       at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>> 2009-02-24 09:37:07,230 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> All 1 .META. region(s) scanned
>> 2009-02-24 09:37:50,642 INFO  
>> org.apache.hadoop.hbase.regionserver.HRegion:
>> Closed .META.,,1
>> 2009-02-24 09:37:56,955 INFO  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> RegionManager.metaScanner scanning meta region {regionname: .META.,, 
>> 1,
>> startKey: <>, server: 127.0.0.1:40076}
>> 2009-02-24 09:37:57,126 ERROR
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed  
>> openScanner
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC  
>> Server
>> handler 2 on 40076, call openScanner([B@155bf4e, [[B@82b436,  
>> [B@1056757,
>> 9223372036854775807, null) from 127.0.0.1
>> :35804: error:  
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>> 2009-02-24 09:38:00,412 WARN  
>> org.apache.hadoop.hbase.master.BaseScanner:
>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>> 127.0.0.1:40076}
>> org.apache.hadoop.hbase.NotServingRegionException:
>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1699)
>>       at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>       at
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>       at
>> org.apache.hadoop.hbase.ipc.HBaseServer 
>> $Handler.run(HBaseServer.java:895)
>>
>>       at  
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>       at
>> sun 
>> .reflect 
>> .NativeConstructorAccessorImpl 
>> .newInstance(NativeConstructorAccessorImpl.java:39)
>>       at
>> sun 
>> .reflect 
>> .DelegatingConstructorAccessorImpl 
>> .newInstance(DelegatingConstructorAccessorImpl.java:27)
>>       at java.lang.reflect.Constructor.newInstance(Constructor.java: 
>> 513)
>>       at
>> org 
>> .apache 
>> .hadoop 
>> .hbase 
>> .RemoteExceptionHandler 
>> .decodeRemoteException(RemoteExceptionHandler.java:95)
>>       at
>> org 
>> .apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java: 
>> 185)
>>       at
>> org 
>> .apache 
>> .hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java: 
>> 73)
>>       at
>> org 
>> .apache 
>> .hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java: 
>> 129)
>>       at
>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java: 
>> 137)
>>       at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>>
>> This just repeats forever, until I restart HBase.
>>
>> Any help would be greatly appreciated.
>> Andrew
>>


Mime
View raw message