hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-964) Startup stuck "waiting for root region"
Date Wed, 05 Nov 2008 18:57:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell updated HBASE-964:
---------------------------------

    Attachment: 964-logs-1.zip

I am also seeing this:

ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root
region
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:793)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:439)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:421)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:449)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:414)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:113)
        at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:310)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:970)
        at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:901)
        at java.lang.Thread.run(Thread.java:619)

I have a very small test master/regionserver setup running latest trunk where I can reliably
reproduce this every time on start up. DEBUG level logs are attached. The logs are large because
I waited until all regions (29) finally came up.

I can test any patches you'd like. 

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region
{regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME
=> '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE =>
{{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME
=> 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH
=> '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME
=> 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION =>
'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE =>
'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region
{regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed
with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate
root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region:
.META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1
in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting
for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed
with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate
root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message