hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3343) Server not shutting down after losing log lease
Date Tue, 21 Dec 2010 01:13:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973438#action_12973438
] 

Jean-Daniel Cryans commented on HBASE-3343:
-------------------------------------------

The issue with TestMetaReaderEditor and what I hope is the same here is that we interrupt
the OpenRegionHandler but don't really handle it inside that thread. Here's an example:

{noformat}
2010-12-20 16:41:53,327 DEBUG [RS_OPEN_REGION-h38.sfo.stumble.net,63068,1292892076786-2]
 handler.OpenRegionHandler(160): Interrupting thread Thread[PostOpenDeployTasks:3d9e86152e5e75258c19d64d4ddb1a4a,5,main]

2010-12-20 16:41:53,327 DEBUG [RS_OPEN_REGION-h38.sfo.stumble.net,63068,1292892076786-2]
 zookeeper.ZKAssign(660): regionserver:63068-0x12d065f78360002 Attempting to transition node
3d9e86152e5e75258c19d64d4ddb1a4a
 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED

2010-12-20 16:41:53,328 WARN  [PostOpenDeployTasks:3d9e86152e5e75258c19d64d4ddb1a4a]
 handler.OpenRegionHandler$PostOpenDeployTasksThread(195):
 Exception running postOpenDeployTasks; region=3d9e86152e5e75258c19d64d4ddb1a4a
java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
...
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
        at $Proxy12.get(Unknown Source)
        at org.apache.hadoop.hbase.catalog.MetaReader.readLocation(MetaReader.java:286)
        at org.apache.hadoop.hbase.catalog.MetaReader.readMetaLocation(MetaReader.java:262)
        at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:279)
        at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:322)
        at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
        at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1331)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:192)
{noformat}

So the region stays opened and the region server will wait on it forever.

> Server not shutting down after losing log lease
> -----------------------------------------------
>
>                 Key: HBASE-3343
>                 URL: https://issues.apache.org/jira/browse/HBASE-3343
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.0
>            Reporter: Todd Lipcon
>            Assignee: stack
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: shutdown-logs.txt.bz2, stuck-server.txt
>
>
> Ran into this bug testing 0.90rc2. I kill -STOPed a server, and then -CONT it after its
logs had been split. It correctly decided it should abort, but got stuck during the shutdown
process.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message