Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CCB6DCC92 for ; Tue, 11 Jun 2013 16:58:22 +0000 (UTC) Received: (qmail 13864 invoked by uid 500); 11 Jun 2013 16:58:21 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 13774 invoked by uid 500); 11 Jun 2013 16:58:21 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 13744 invoked by uid 99); 11 Jun 2013 16:58:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jun 2013 16:58:20 +0000 Date: Tue, 11 Jun 2013 16:58:20 +0000 (UTC) From: "stack (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-8687) When moving region with region_mover.rb, there is long stack trace for RegionMovedException MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-8687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13680477#comment-13680477 ] stack commented on HBASE-8687: ------------------------------ Looking in code, the RegionMovedException uses a cache of regions recently moved to point out where the region has gone too. The server above that threw the exception was or was not hor8n09? If it was, then that is odd. If region is still on this server, we should be fixing up the recently moved cache. > When moving region with region_mover.rb, there is long stack trace for RegionMovedException > ------------------------------------------------------------------------------------------- > > Key: HBASE-8687 > URL: https://issues.apache.org/jira/browse/HBASE-8687 > Project: HBase > Issue Type: Bug > Components: Region Assignment > Reporter: Ted Yu > Priority: Minor > > When gracefully rolling restart region servers, I saw the following in output: > {code} > 2013-06-04 20:44:40,135 DEBUG [main] client.ClientScanner: Scan table=usertable, startRow=user8129671889902366092 > 2013-06-04 20:44:40,141 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=usertable,user8129671889902366092,00000000000000 > 2013-06-04 20:44:40,158 INFO [main] region_mover: Moving region 13168d8b86f1ace9472f60555207a707 (2 of 2) to server=hor8n09.gq1.ygridcore.net,60020,1370378675859 > 2013-06-04 20:44:40,405 DEBUG [main] client.ClientScanner: Scan table=usertable, startRow=user8129671889902366092 > 2013-06-04 20:44:40,407 WARN [main] client.ServerCallable: Call exception, tries=0, numRetries=100 > org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=hor8n09.gq1.ygridcore.net port=60020 startCode=1370378675859. As of locationSeqNum=194375. > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:230) > at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:299) > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:147) > at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:55) > at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:174) > at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:215) > at org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:130) > at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:585) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:450) > at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:311) > at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:59) > at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:167) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.method__6$RUBY$isSuccessfulScan(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb:121) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__6$RUBY$isSuccessfulScan.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__6$RUBY$isSuccessfulScan:65535) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__6$RUBY$isSuccessfulScan.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__6$RUBY$isSuccessfulScan:65535) > at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.method__8$RUBY$move(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb:164) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__8$RUBY$move.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__8$RUBY$move:65535) > at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:181) > at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:69) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.block_6$RUBY$__for__(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb:381) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$block_6$RUBY$__for__.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$block_6$RUBY$__for__:65535) > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:112) > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:95) > at org.jruby.runtime.Block.yield(Block.java:130) > at org.jruby.ast.YieldNode.interpret(YieldNode.java:119) > at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) > at org.jruby.ast.WhileNode.interpret(WhileNode.java:131) > at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) > at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) > at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) > at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169) > at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191) > at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302) > at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144) > at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.method__27$RUBY$loadRegions(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb:363) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__27$RUBY$loadRegions.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__27$RUBY$loadRegions:65535) > at homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__27$RUBY$loadRegions.call(homes$hortonzy$hbase_minus_0_dot_95_dot_1$bin$region_mover$method__27$RUBY$loadRegions:65535) > at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332) > at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.__file__(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb:454) > at homes.hortonzy.hbase_minus_0_dot_95_dot_1.bin.region_mover.load(/homes/hortonzy/hbase-0.95.1/bin/region_mover.rb) > at org.jruby.Ruby.runScript(Ruby.java:697) > at org.jruby.Ruby.runScript(Ruby.java:690) > at org.jruby.Ruby.runNormally(Ruby.java:597) > at org.jruby.Ruby.runFromMain(Ruby.java:446) > at org.jruby.Main.doRunFromMain(Main.java:369) > at org.jruby.Main.internalRun(Main.java:258) > at org.jruby.Main.run(Main.java:224) > at org.jruby.Main.run(Main.java:208) > at org.jruby.Main.main(Main.java:188) > Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.exceptions.RegionMovedException): org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=hor8n09.gq1.ygridcore.net port=60020 startCode=1370378675859. As of locationSeqNum=194375. > at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2550) > at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3844) > at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:2997) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:20932) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2122) > at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1829) > at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1336) > at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1540) > at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1597) > at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:21367) > at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:290) > ... 54 more > 2013-06-04 20:44:40,912 DEBUG [main] client.HConnectionManager$HConnectionImplementation: Removed hor8n02.gq1.ygridcore.net:60020 as a location of usertable,user8129671889902366092,1370373946456.13168d8b86f1ace9472f60555207a707. for tableName=usertable from cache > Reloaded hor8n09.gq1.ygridcore.net region(s) > {code} > The script tried to move region to hor8n09. > RegionMovedException involving hor8n09 is misleading because hor8n09 is the recipient of the region movement. > Looks like such exception should be suppressed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira