Return-Path: Delivered-To: apmail-hadoop-hbase-user-archive@minotaur.apache.org Received: (qmail 53498 invoked from network); 26 Feb 2009 17:21:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 26 Feb 2009 17:21:05 -0000 Received: (qmail 9493 invoked by uid 500); 26 Feb 2009 17:21:03 -0000 Delivered-To: apmail-hadoop-hbase-user-archive@hadoop.apache.org Received: (qmail 9478 invoked by uid 500); 26 Feb 2009 17:21:03 -0000 Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-user@hadoop.apache.org Delivered-To: mailing list hbase-user@hadoop.apache.org Received: (qmail 9465 invoked by uid 99); 26 Feb 2009 17:21:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 09:21:03 -0800 X-ASF-Spam-Status: No, hits=-8.0 required=10.0 tests=RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.212 as permitted sender) Received: from [131.107.115.212] (HELO smtp.microsoft.com) (131.107.115.212) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 17:20:51 +0000 Received: from TK5-EXHUB-C102.redmond.corp.microsoft.com (157.54.18.53) by TK5-EXGWY-E801.partners.extranet.microsoft.com (10.251.56.50) with Microsoft SMTP Server (TLS) id 8.2.99.4; Thu, 26 Feb 2009 09:20:21 -0800 Received: from NA-EXMSG-C103.redmond.corp.microsoft.com ([157.54.110.52]) by TK5-EXHUB-C102.redmond.corp.microsoft.com ([157.54.18.53]) with mapi; Thu, 26 Feb 2009 09:20:21 -0800 From: "Jim Kellerman (POWERSET)" To: "hbase-user@hadoop.apache.org" Date: Thu, 26 Feb 2009 09:20:18 -0800 Subject: RE: Hbase stops working Thread-Topic: Hbase stops working Thread-Index: AcmYK5m5AIdgdvoqSf2KQjfsjjce0AAClCiA Message-ID: References: <2957AA2E-1D2C-4742-B03C-6E44A60B4256@goroam.net> <31a243e70902260457gb6a9678je5e97472621bb05b@mail.gmail.com> <31a243e70902260541w1202a083nba336f68a44413d2@mail.gmail.com> <2E3609C3-3CF0-41CC-AD31-E27379351CE8@goroam.net> <31a243e70902260800nf5c01d3n1535e18c1a442cb7@mail.gmail.com> In-Reply-To: <31a243e70902260800nf5c01d3n1535e18c1a442cb7@mail.gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org With a single core machine, I'm amazed it works at all. Between the namenode, data node(s), job tracker, task tracker, hbase master and regionserver(s), that's a whole lot of threads (on the order of several hundred). Thread starvation is a real possibility as the jvm makes no guarantees about which runnable thread gets the cpu next. --- Jim Kellerman, Powerset (Live Search, Microsoft Corporation) > -----Original Message----- > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean- > Daniel Cryans > Sent: Thursday, February 26, 2009 8:01 AM > To: hbase-user@hadoop.apache.org > Subject: Re: Hbase stops working > > Andrew, > > At 512MB I'm sure it swapped, tho it would be hard to tell now unless you > installed some monitoring software. Next time, you should at least monito= r > your job using top to make sure it's the root cause of the problem. > > Something that may help you is in > http://wiki.apache.org/hadoop/Hbase/Troubleshooting#7 mainly the third > bullet point in Resolution (adding -XX:+UseConcMarkSweepGC in hbase-env.s= h > in the HBASE_OPTS and uncommenting it). > > J-D > > On Thu, Feb 26, 2009 at 10:14 AM, Andrew McCall > wrote: > > > Hi J-D, > > > > I'm running the HBase 0.19 release downloaded to the website on a prett= y > > lacklustre linux machine; 1.6Ghz single core processor, 512MB ram with > 1.5GB > > swap space enabled and 80GB hard disk. It's a testing machine, I've bee= n > > using for development. It's running Hadoop core 0.19 in pseudo > distributed > > mode and running a map/reduce reduce crawl using nutch and HBase. I've > been > > planning to move to an EC3 cluster in the near future but I've been > putting > > it off since everything seemed to come up and work on the testing box. > > > > It's entirely possible that it is a result of a swap, how do I tell? > I've > > looked in all the logs for hadoop, hbase and my job and can't see that > > anything stopped logging around then, the logs all have entries for > 9:34. > > I've also increased my lease timeout from the default to 600,000 ms in > my > > hbase-site, which is below as a result of another issue. I did search o= f > the > > mailing list for the other issue and answers suggested that the lease > > timeout should be the same as my map/reduce job timeout. > > > > > > > > hbase.rootdir > > hdfs://localhost:9000/hbase > > The directory shared by region servers. > > > > > > > > hbase.regionserver.lease.period > > 600000 > > > > > > > > Thanks for the help, > > Andrew > > > > > > > > On 26 Feb 2009, at 13:41, Jean-Daniel Cryans wrote: > > > > Andrew, > >> > >> Oh yes sorry, your setup is in local mode so no region server in a > >> different > >> thread. My bad! > >> > >> That lease timeout probably means that you were swapping and a full jv= m > >> garbage collecting occurred which blocked the main thread. Can you mak= e > >> sure > >> that it did swap at that moment? > >> > >> It would also be helpful to get the usual info like hardware, OS, hbas= e > >> version, etc. > >> > >> Thx! > >> > >> J-D > >> > >> On Thu, Feb 26, 2009 at 8:35 AM, Andrew McCall > >> >wrote: > >> > >> Hi J-D, > >>> > >>> I did check the region server logs and they didn't seem to be very > useful > >>> at all: > >>> > >>> Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.ne= t > >>> ulimit -n 1024 > >>> 2009-02-23 22:11:58,817 WARN > >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a > >>> distinct > >>> region server because hbase.master is set to 'local' mode > >>> > >>> Is all that is in them, once for each time I started HBase. > >>> > >>> I saw the shutdown message, which was preceded by this: > >>> > >>> 2009-02-24 09:34:41,579 INFO > >>> org.apache.hadoop.hbase.master.ServerManager: > >>> 127.0.0.1:40076 lease expired > >>> > >>> I can't find any evidence it's related to my map/reduce task, it's th= e > >>> only > >>> one in the log files and it's suspiciously just before the process > >>> shutdown > >>> message - but, I'm new to this and honestly don't know what to make o= f > >>> it. > >>> > >>> Andrew > >>> > >>> > >>> On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote: > >>> > >>> Andrew, > >>> > >>>> > >>>> For some reason, your region server shut down around 9:36: > >>>> > >>>> 2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase. > >>>> > >>>> > >>>>> master.RegionServerOperation: process shutdown of server > >>>>> 127.0.0.1:40076 > >>>>> : > >>>>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1, > >>>>> > >>>>> > >>>> > >>>> But just before that the master received a startup message from it. > Did > >>>> you > >>>> take a look at your region server log? What does it say around that > >>>> time? > >>>> > >>>> Thx > >>>> > >>>> J-D > >>>> > >>>> On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall < > >>>> andrew.mccall@goroam.net > >>>> > >>>>> wrote: > >>>>> > >>>> > >>>> I've been testing a nuch on HBase and I'm running into an issue with > it > >>>> > >>>>> when it's been running for a while. > >>>>> > >>>>> I'm running hbase 0.19 on a single local host pseudo distributed > hadoop > >>>>> 0.19 cluster. > >>>>> > >>>>> After some time I get the following in my logs: > >>>>> > >>>>> 2009-02-24 09:31:18,664 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:31:18,664 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:31:24,293 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region > {regionname: > >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:31:24,293 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT- > ,,0, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:31:27,217 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region > {regionname: > >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:31:27,217 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:31:55,533 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region > {regionname: > >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:32:18,665 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:32:35,749 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT- > ,,0, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:32:38,957 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region > {regionname: > >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:32:38,957 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:33:18,771 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:33:54,889 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region > {regionname: > >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:33:57,387 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT- > ,,0, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:34:41,579 INFO > >>>>> org.apache.hadoop.hbase.master.ServerManager: > >>>>> 127.0.0.1:40076 lease expired > >>>>> 2009-02-24 09:34:53,440 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region > {regionname: > >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:34:53,441 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:34:53,441 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:35:38,720 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region > {regionname: > >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:35:38,720 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:35:56,951 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:36:05,667 INFO > >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: > >>>>> MSG_CALL_SERVER_STARTUP > >>>>> 2009-02-24 09:36:05,668 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.rootScanner scan of 1 row(s) of meta region > {regionname: > >>>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:36:05,884 INFO > >>>>> org.apache.hadoop.hbase.master.RegionServerOperation: process > shutdown > >>>>> of > >>>>> server 127.0.0.1:40076: logSplit: false, rootRescanned: false, > >>>>> numberOfMetaRegions: 1, > >>>>> onlineMetaRegions.size(): 1 > >>>>> 2009-02-24 09:36:17,689 INFO > org.apache.hadoop.hbase.regionserver.HLog: > >>>>> Splitting 1 log(s) in > >>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 > >>>>> 2009-02-24 09:36:25,034 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scan of 2 row(s) of meta region > {regionname: > >>>>> .META.,,1, startKey: <>, server: 127.0.0.1:40 > >>>>> 076} complete > >>>>> 2009-02-24 09:36:25,034 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:36:36,598 INFO > org.apache.hadoop.hbase.regionserver.HLog: > >>>>> log > >>>>> file splitting completed for > >>>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076 > >>>>> 2009-02-24 09:36:56,953 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:37:05,279 ERROR > >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed > openScanner > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >>>>> Server > >>>>> handler 4 on 40076, call openScanner([B@1ac5a78, [[B@6799dc, > >>>>> [B@1a66cfc, > >>>>> 9223372036854775807, null) from 127.0.0.1 > >>>>> :50139: error: org.apache.hadoop.hbase.NotServingRegionException: > >>>>> .META.,,1 > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> 2009-02-24 09:37:07,051 WARN > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server: > >>>>> 127.0.0.1:40076} > >>>>> org.apache.hadoop.hbase.NotServingRegionException: > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> > >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>>>> Method) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc= c > essorImpl.java:39) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst= r > uctorAccessorImpl.java:27) > >>>>> at > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remo= t > eExceptionHandler.java:95) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:18= 5 > ) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.= j > ava:73) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.ja= v > a:129) > >>>>> at > >>>>> > org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) > >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:65) > >>>>> 2009-02-24 09:37:07,230 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> All 1 .META. region(s) scanned > >>>>> 2009-02-24 09:37:50,642 INFO > >>>>> org.apache.hadoop.hbase.regionserver.HRegion: > >>>>> Closed .META.,,1 > >>>>> 2009-02-24 09:37:56,955 INFO > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> RegionManager.metaScanner scanning meta region {regionname: > .META.,,1, > >>>>> startKey: <>, server: 127.0.0.1:40076} > >>>>> 2009-02-24 09:37:57,126 ERROR > >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed > openScanner > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >>>>> Server > >>>>> handler 2 on 40076, call openScanner([B@155bf4e, [[B@82b436, > >>>>> [B@1056757, > >>>>> 9223372036854775807, null) from 127.0.0.1 > >>>>> :35804: error: org.apache.hadoop.hbase.NotServingRegionException: > >>>>> .META.,,1 > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> 2009-02-24 09:38:00,412 WARN > >>>>> org.apache.hadoop.hbase.master.BaseScanner: > >>>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server: > >>>>> 127.0.0.1:40076} > >>>>> org.apache.hadoop.hbase.NotServingRegionException: > >>>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe= r > .java:2065) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionSer= v > er.java:1699) > >>>>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= m > pl.java:25) > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>> at > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632) > >>>>> at > >>>>> > >>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895) > >>>>> > >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>>>> Method) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc= c > essorImpl.java:39) > >>>>> at > >>>>> > >>>>> > >>>>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst= r > uctorAccessorImpl.java:27) > >>>>> at > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(Remo= t > eExceptionHandler.java:95) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:18= 5 > ) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.= j > ava:73) > >>>>> at > >>>>> > >>>>> > >>>>> > org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.ja= v > a:129) > >>>>> at > >>>>> > org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137) > >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:65) > >>>>> > >>>>> This just repeats forever, until I restart HBase. > >>>>> > >>>>> Any help would be greatly appreciated. > >>>>> Andrew > >>>>> > >>>>> > >>>>> > >>> > >