hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5356) region_mover.rb can hang if table region it belongs to is deleted.
Date Thu, 09 Feb 2012 00:04:57 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204137#comment-13204137
] 

Jonathan Hsieh commented on HBASE-5356:
---------------------------------------

rephrase of previous comment: The region_mover script initially caches a list of regions to
move. If the table is deleted after the cached list is gathered but before all regions are
moved, the script can get stuck attempting to move a deleted region.  

In a related but likely separate issue -- if a new presplit table is created, as the region_mover
is emptying an RS, the emptying RS is a candidate for new regions and will get some of the
new regions.  Ideally this is fenced off so that it does not get regions assigned to it, but
this would require some ZK.  This seems less important because the majority of regions will
be moved off the RS and the few new regions can rely on the automatically fail over to other
RS's. 


                
> region_mover.rb can hang if table region it belongs to is deleted.
> ------------------------------------------------------------------
>
>                 Key: HBASE-5356
>                 URL: https://issues.apache.org/jira/browse/HBASE-5356
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.3, 0.94.0, 0.92.0
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>            Priority: Minor
>
> I was testing the region_mover.rb script on a loaded hbase and noticed that it can hang
(thus hanging graceful shutdown) if a region that it is attempting to move gets deleted (by
a table delete operation).
> Here's the start of the relevent stack dump
> {code}
> 12/02/08 13:27:13 WARN client.HConnectionManager$HConnectionImplementation: Encountered
problems when prefetch META table:
> org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table:
TestLoadAndVerify_1328735001040, row=TestLoadAnd\
> Verify_1328735001040,yC^P\xD7\x945\xD4,99999999999999
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:136)
>         at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:95)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:64\
> 9)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:703\
> )
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:565)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
>         at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
>         at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.\
> java:1018)
>         at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
>         at org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
>         at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
>         at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:525)
>         at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:380)
>         at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:58)
>         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
>         at usr.lib.hbase.bin.region_mover.method__7$RUBY$isSuccessfulScan(/usr/lib/hbase/bin/region_mover.rb:133)
>         at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
> sfulScan:65535)
>         at usr$lib$hbase$bin$region_mover#method__7$RUBY$isSuccessfulScan.call(usr$lib$hbase$bin$region_mover#method__7$RUBY$isSucces\
> sfulScan:65535)
>         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:171)
>         at usr.lib.hbase.bin.region_mover.block_4$RUBY$__for__(/usr/lib/hbase/bin/region_mover.rb:326)
>         at usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__.call(usr$lib$hbase$bin$region_mover#block_4$RUBY$__for__:65535)
>         at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:133)
>         at org.jruby.runtime.BlockBody.call(BlockBody.java:73)
>         at org.jruby.runtime.Block.call(Block.java:89)
>         at org.jruby.RubyProc.call(RubyProc.java:268)
>         at org.jruby.RubyProc.call(RubyProc.java:228)
>         at org.jruby.RubyProc$i$0$0$call.call(RubyProc$i$0$0$call.gen:65535)
>         at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:209)
>         at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
>         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:137)
>         at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
>         at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:103)
>         at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
>         at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
>         at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
>         at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:171)
>         at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:272)
>         at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:114)
>         at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:123)
>         at usr.lib.hbase.bin.region_mover.chained_26_rescue_4$RUBY$SYNTHETICunloadRegions(/usr/lib/hbase/bin/region_mover.rb:319)
>         at usr.lib.hbase.bin.region_mover.method__25$RUBY$unloadRegions(/usr/lib/hbase/bin/region_mover.rb:313)
>         at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
>         at usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions.call(usr$lib$hbase$bin$region_mover#method__25$RUBY$unloadRegions:65535)
>         at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
>         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:173)
>         at usr.lib.hbase.bin.region_mover.__file__(/usr/lib/hbase/bin/region_mover.rb:430)
>         at usr.lib.hbase.bin.region_mover.load(/usr/lib/hbase/bin/region_mover.rb)
>         at org.jruby.Ruby.runScript(Ruby.java:670)
>         at org.jruby.Ruby.runNormally(Ruby.java:574)
>         at org.jruby.Ruby.runFromMain(Ruby.java:423)
>         at org.jruby.Main.doRunFromMain(Main.java:278)
>         at org.jruby.Main.internalRun(Main.java:198)
>         at org.jruby.Main.run(Main.java:164)
>         at org.jruby.Main.run(Main.java:148)
>         at org.jruby.Main.main(Main.java:128)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message