hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nickolai Toupikov <nickolai.toupi...@deri.org>
Subject Issue with NotServingRegionException
Date Tue, 05 Aug 2008 12:47:54 GMT

Hello!
It seems I have a serious problem with HBase. We've been using if for 
some time now, had some minor problems with regions unexpectedly closing 
once every couple of days, but this would usually sort itself out after 
a couple of stops and starts. The hbase runs atop of a cluster of 3 
machines. One of them is running HMaster and regionserver the other two 
are regionservers .The replication is 3.
So while after restarting hbase this morning, I realize I cant get any 
record out of it - a 'get' for a random record in hbase shell returns this:

                hbase(main):001:0>  get 'page-repository',
        'http://sws.geonames.org/3623940/'
                NativeException:
        org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying
        to contact region server 140.203.154.174:60020 for region
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058,
        row 'http://sws.geonames.org/3623940/', but failed after 3 attempts.
                Exceptions:
                org.apache.hadoop.hbase.NotServingRegionException:
        org.apache.hadoop.hbase.NotServingRegionException:
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                    at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:597)
                    at
        org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
                    at
        org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

                org.apache.hadoop.hbase.NotServingRegionException:
        org.apache.hadoop.hbase.NotServingRegionException:
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                    at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:597)
                    at
        org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
                    at
        org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

                org.apache.hadoop.hbase.NotServingRegionException:
        org.apache.hadoop.hbase.NotServingRegionException:
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                    at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:597)
                    at
        org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
                    at
        org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)


                    from
        org/apache/hadoop/hbase/client/HConnectionManager.java:885:in
        `getRegionServerWithRetries'
                    from
        org/apache/hadoop/hbase/client/HTable.java:546:in `getRow'
                    from
        org/apache/hadoop/hbase/client/HTable.java:507:in `getRow'
                    from sun/reflect/NativeMethodAccessorImpl.java:-2:in
        `invoke0'
                    from sun/reflect/NativeMethodAccessorImpl.java:39:in
        `invoke'
                    from
        sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
                    from java/lang/reflect/Method.java:597:in `invoke'
                    from org/jruby/javasupport/JavaMethod.java:250:in
        `invokeWithExceptionHandling'
                    from org/jruby/javasupport/JavaMethod.java:219:in
        `invoke'
                    from org/jruby/javasupport/JavaClass.java:416:in
        `execute'
                    from
        org/jruby/internal/runtime/methods/SimpleCallbackMethod.java:67:in
        `call'
                    from
        org/jruby/internal/runtime/methods/DynamicMethod.java:78:in `call'
                    from org/jruby/runtime/CallSite.java:155:in
        `cacheAndCall'
                    from org/jruby/runtime/CallSite.java:332:in `call'
                    from org/jruby/evaluator/ASTInterpreter.java:649:in
        `callNode'
                    from org/jruby/evaluator/ASTInterpreter.java:324:in
        `evalInternal'
                ... 129 levels...
                    from
        ruby.sindice.hbase_minus_0_dot_2_dot_0.bin.hirbInvokermethod__22$RUBY$startOpt:-1:in
        `call'
                    from
        org/jruby/internal/runtime/methods/DynamicMethod.java:74:in `call'
                    from
        org/jruby/internal/runtime/methods/CompiledMethod.java:48:in `call'
                    from org/jruby/runtime/CallSite.java:123:in
        `cacheAndCall'
                    from org/jruby/runtime/CallSite.java:298:in `call'
                    from
        ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:338:in
        `__file__'
                    from
        ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:-1:in
        `__file__'
                    from
        ruby/sindice/hbase_minus_0_dot_2_dot_0/bin//home/sindice/hbase/bin/../bin/hirb.rb:-1:in
        `load'
                    from org/jruby/Ruby.java:512:in `runScript'
                    from org/jruby/Ruby.java:432:in `runNormally'
                    from org/jruby/Ruby.java:312:in `runFromMain'
                    from org/jruby/Main.java:144:in `run'
                    from org/jruby/Main.java:89:in `run'
                    from org/jruby/Main.java:80:in `main'
                    from /home/sindice/hbase/bin/../bin/hirb.rb:255:in `get'
                    from (hbase):2:in `binding'




At the Same Time, one  regionserver (which one depends on the requested 
field i guess) showers his logs with this sort of messages

                2008-08-05 13:33:51,768 INFO
        org.apache.hadoop.ipc.Server: IPC Server handler 8 on 60020,
        call getRow([B@f6d64c5, [B@56f2c96c, 9223372036854775807) from
        140.203.154.174:35053: error:
        org.apache.hadoop.hbase.NotServingRegionException:
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
                org.apache.hadoop.hbase.NotServingRegionException:
        page-repository,http://sws.geonames.org/3360219/nearby.rdf,1215824259058
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:1436)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1077)
                    at
        org.apache.hadoop.hbase.regionserver.HRegionServer.getRow(HRegionServer.java:1053)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                    at
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                    at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                    at java.lang.reflect.Method.invoke(Method.java:597)
                    at
        org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
                    at
        org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)



What can I do to get rid of this problem and have HBase get back to work?

Thank you very much for your time,
Regards,
Nickolai

Mime
View raw message