hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2251) [hbase] master won't go down if root region was not found
Date Thu, 22 Nov 2007 05:45:43 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HADOOP-2251:
--------------------------

    Component/s: contrib/hbase

Looks like just saw another incidence of this internally but hard to tell because DEBUG log
level not enabled.... master wouldn't go down... was complaining that it couldn't talk to
the server w/ ROOT:

{code}
2007-11-22 05:02:43,504 INFO org.apache.hadoop.hbase.HMaster: HMaster.rootScanner scanning
meta region regionname: -ROOT-,,0, startKey: <>, server: 208.76.45.216:60020}
2007-11-22 05:02:47,346 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanning
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020}
2007-11-22 05:02:47,532 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scan of
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020} complete
2007-11-22 05:02:47,532 INFO org.apache.hadoop.hbase.HMaster: all meta regions scanned
2007-11-22 05:03:43,508 WARN org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
        at org.apache.hadoop.ipc.Client.call(Client.java:484)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at $Proxy2.openScanner(Unknown Source)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
        at org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505) 
        at org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
2007-11-22 05:03:47,347 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanning
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020}
2007-11-22 05:03:47,530 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scan of
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020} complete
2007-11-22 05:03:47,530 INFO org.apache.hadoop.hbase.HMaster: all meta regions scanned
2007-11-22 05:03:53,511 INFO org.apache.hadoop.hbase.HMaster: HMaster.rootScanner scanning
meta region regionname: -ROOT-,,0, startKey: <>, server: 208.76.45.216:60020}
2007-11-22 05:04:47,349 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanning
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020}
2007-11-22 05:04:47,534 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scan of
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020} complete
2007-11-22 05:04:47,534 INFO org.apache.hadoop.hbase.HMaster: all meta regions scanned
2007-11-22 05:04:53,512 ERROR org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
        at org.apache.hadoop.ipc.Client.call(Client.java:484)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at $Proxy2.openScanner(Unknown Source)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
        at org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505)
        at org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
2007-11-22 05:05:03,514 INFO org.apache.hadoop.hbase.HMaster: HMaster.rootScanner scanning
meta region regionname: -ROOT-,,0, startKey: <>, server: 208.76.45.216:60020}
2007-11-22 05:05:47,351 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scanning
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020}
2007-11-22 05:05:47,531 INFO org.apache.hadoop.hbase.HMaster: HMaster.metaScanner scan of
meta region regionname: .META.,,1, startKey: <>, server: 208.76.45.131:60020} complete
2007-11-22 05:05:47,531 INFO org.apache.hadoop.hbase.HMaster: all meta regions scanned
2007-11-22 05:06:03,515 ERROR org.apache.hadoop.hbase.HMaster: Scan ROOT region
java.net.SocketTimeoutException: timed out waiting for rpc response
        at org.apache.hadoop.ipc.Client.call(Client.java:484)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
        at $Proxy2.openScanner(Unknown Source)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(HMaster.java:216)
        at org.apache.hadoop.hbase.HMaster$RootScanner.scanRoot(HMaster.java:505)
        at org.apache.hadoop.hbase.HMaster$RootScanner.maintenanceScan(HMaster.java:548)
        at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java:197)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:58)
....
{code}

..and so on for ever.  Why ain't we noticing the ROOT has gone bad?

> [hbase] master won't go down if root region was not found
> ---------------------------------------------------------
>
>                 Key: HADOOP-2251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2251
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>            Reporter: stack
>            Priority: Minor
>
> psaab couldn't shut down his master... was getting reams of the below:
> {code}
> 2007-11-21 13:45:30,421 INFO  hbase.HMaster - process shutdown of server 38.99.76.15:60020:
logSplit: true, rootChecked: true, rootRescanned: false, numberOfMetaRegions: 0, onlineMetaRegions.size():
0
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - process server shutdown scanning root region
cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - Put PendingServerShutdown of 38.99.76.15:60020
back on queue
> 2007-11-21 13:45:30,421 DEBUG hbase.HMaster - Main processing loop: PendingServerShutdown
of 38.99.76.21:60020
> 2007-11-21 13:45:30,422 INFO  hbase.HMaster - process shutdown of server 38.99.76.21:60020:
logSplit: true, rootChecked: true, rootRescanned: false, numberOfMetaRegions: 0, onlineMetaRegions.size():
0
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - process server shutdown scanning root region
cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Put PendingServerShutdown of 38.99.76.21:60020
back on queue
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Main processing loop: PendingServerShutdown
of 38.99.76.31:60020
> 2007-11-21 13:45:30,422 INFO  hbase.HMaster - process shutdown of server 38.99.76.31:60020:
logSplit: true, rootChecked: true, rootRescanned: false, numberOfMetaRegions: 0, onlineMetaRegions.size():
0
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - process server shutdown scanning root region
cancelled because rootRegionLocation is null
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Put PendingServerShutdown of 38.99.76.31:60020
back on queue
> 2007-11-21 13:45:30,422 DEBUG hbase.HMaster - Main processing loop: PendingServerShutdown
of 38.99.76.17:60020
> ..
> {code}
> Looks like a shutdown soon after startup so should be reproducible.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message