hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: HRegionServer: Failed openScanner
Date Fri, 15 May 2009 17:18:47 GMT
The region server hosting META could not communicate with the master for a very long time.
Some kind of network issue? Any entries in the region server logs above this one

> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying

which may be relevant? Anything about sleeping too long? 

Related, there were some bugs that I am aware of preventing recovery if META in particular
goes away but they should be fixed for 0.20 as of https://issues.apache.org/jira/browse/HBASE-1362
.

   - Andy






________________________________
From: Sasha Dolgy <sdolgy@gmail.com>
To: hbase-user@hadoop.apache.org
Sent: Friday, May 15, 2009 9:33:24 AM
Subject: Re: HRegionServer: Failed openScanner

on a devel system i don't have any monitoring in place.  i just leave it and
hope the log files can give me hints when it breaks.  in the master log i
see the log info below.  the machine wasn't under any heavy work ... been
the same amount of load for the past 48 hours or so.  for the tests
everything is on one machine...
2009-05-15 00:52:49,186 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:52:49,336 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:54:43,959 INFO org.apache.hadoop.hbase.master.ServerManager:
127.0.0.1:60020 lease expired
2009-05-15 00:54:47,417 INFO
org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
server 127.0.0.1:60020: logSplit: false, rootRescanned: false,
numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
2009-05-15 00:54:50,049 INFO org.apache.hadoop.hbase.regionserver.HLog:
Splitting 27 log(s) in hdfs://
foo.bar.net:9000/hbase/log_127.0.0.1_1242258287822_60020
2009-05-15 00:55:54,866 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
-ROOT-,,0, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 11 row(s) of meta region {regionname:
.META.,,1, startKey: <>, server: 127.0.0.1:60020} complete
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:55:57,246 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,server: 127.0.0.1:60020}
2009-05-15 00:55:57,727 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
        at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
        at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
        at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
        at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
2009-05-15 00:55:57,977 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-05-15 00:56:57,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startKey: <>,
server: 127.0.0.1:60020}
2009-05-15 00:57:02,214 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {regionname: .META.,,1, startKey: <>, server:
127.0.0.1:60
020}
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
        at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
        at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
        at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)


On Fri, May 15, 2009 at 5:13 PM, Andrew Purtell <apurtell@apache.org> wrote:

> > 2009-05-15 00:55:53,090 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> > master for 189261 milliseconds - retrying
>
> What do you see in the master log around this time?
>
> Was your cluster heavily loaded and/or in swap at this time? Do you have
> monitoring in place (i.e. sar (sysstat), Ganglia, Nagios) where you can go
> back in time and look at what was going on at the OS or network level at the
> time?
>
> Best regards,
>
>   - Andy
>
>
>
>
> ________________________________
> From: Sasha Dolgy <sasha.dolgy@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Friday, May 15, 2009 1:53:37 AM
> Subject: HRegionServer: Failed openScanner
>
> Hi there,
>
> Thought I would run a count 'table-name' to see how many records are in a
> table.  The count  didn't work so I went and took a look in the
> regionserver
> log file and see the below.  Now, to be honest, not quite sure why it's
> just
> stopped working.  The lines are from the start of the log file (from may
> 15).  The log file from May 14 has no errors in it.  Is there any way I can
> enable debugging to find out why it's stopped working?  I had added this in
> the past 36 hours but it didn't appear to cause any problems and was
> working
> fine for atleast 24 hours:
>
>  <property>
>        <name>hbase.regionserver.class</name>
>        <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
>        <description>enable indexing</description>
>  </property>
>
>  <property>
>        <name>hbase.regionserver.impl</name>
>
>
> <value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
>        <description>enable indexing</description>
>  </property>
>
>
> $HBASE_HOME/bin/stop-all.sh and $HBASE_HOME/bin/start-all.sh fixed
> everything and it's all working fine again.
>
> 2009-05-15 00:55:53,090 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to
> master for 189261 milliseconds - retrying
> 2009-05-15 00:55:56,789 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> MSG_CALL_SERVER_STARTUP:
> safeMode=false
> 2009-05-15 00:55:57,249 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:55:57,252 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020, call openScanner([B@233dfbc1, [[B@3a5b4dfa,
> [B@405c7604,
> 9223372036854775807, null) from 127.0.0.1:47277: error:
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
> 2009-05-15 00:56:57,751 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2076)
>        at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1710)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)
>
> --
> Sasha Dolgy
> sasha.dolgy@gmail.com
>
>
>
>
>



-- 
Sasha Dolgy
sasha.dolgy@gmail.com



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message