hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Kirichenko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
Date Thu, 31 Oct 2013 13:37:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810240#comment-13810240
] 

Sergey Kirichenko commented on HBASE-8912:
------------------------------------------

May be this helps (HBase from cloudera - 0.94.6-cdh4.4.0):

grep by region caused exception on master:
{noformat}
2013-10-31 00:07:52,871 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3a476d37da81f620a3e53179d7d9192b
has null regionLocation. But its table table_x isn't in ENABLING state.
2013-10-31 00:07:53,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070
Async create of unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:53,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback:
rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=OFFLINE, ts=1383163673057, server=null, server=xxx100,60020,1383163665902
2013-10-31 00:07:53,495 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback:
rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=OFFLINE, ts=1383163673057, server=null
2013-10-31 00:07:54,834 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING,
server=xxx100,60020,1383163665902, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx100,60020,1383163665902, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing
plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx100,60020,1383163665902
2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx108,60020,1383163666006; 9 (online=9, available=8) available servers
2013-10-31 00:07:56,955 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler:
Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163675624, server=xxx100,60020,1383163665902
2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070
Creating (or updating) unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing
plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx108,60020,1383163666006
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing
plan for region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.;
plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx108,60020,1383163666006
2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning
region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
to xxx108,60020,1383163666006
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing
plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx108,60020,1383163666006
2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx106,60020,1383163666003; 9 (online=9, available=8) available servers
2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler:
Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006
2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070
Creating (or updating) unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state
2013-10-31 00:07:58,553 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN,
server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing
plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx106,60020,1383163666003
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous
transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx104,60020,1383163665976; 9 (online=9, available=8) available servers
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler:
Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b
2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing
plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
destination server is xxx104,60020,1383163665976
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing
plan for region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.;
plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
src=, dest=xxx104,60020,1383163665976
2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning
region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
to xxx104,60020,1383163665976
2013-10-31 00:07:58,595 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 .. Cannot transit
it to OFFLINE.
java.lang.IllegalStateException: Unexpected state : table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 .. Cannot transit
it to OFFLINE.
        at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1831)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1661)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
        at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
{noformat}


grep by region caused exception on xxx100:
{noformat}
2013-10-31 00:07:54,000 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:54,000 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:54,029 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
{NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:55,447 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:55,501 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:55,546 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:55,602 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:55,613 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:55,618 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:55,621 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:55,621 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:55,630 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING
to RS_ZK_REGION_FAILED_OPEN
{noformat}

grep by region caused exception on xxx108:
{noformat}
2013-10-31 00:07:57,003 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:57,010 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:57,042 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
{NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:57,049 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:57,060 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:57,065 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:57,095 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:57,105 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:57,107 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:57,108 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:57,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:57,125 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING
to RS_ZK_REGION_FAILED_OPEN
{noformat}

grep by region caused exception on xxx104:
{noformat}
2013-10-31 00:07:58,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received
request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:58,587 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:58,602 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2013-10-31 00:07:58,603 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region:
{NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,}
2013-10-31 00:07:58,604 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.
2013-10-31 00:07:58,610 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b
is a link
2013-10-31 00:07:58,621 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true
2013-10-31 00:07:58,627 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833
is a link
2013-10-31 00:07:58,639 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8
is a link
2013-10-31 00:07:58,650 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8,
isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false
2013-10-31 00:07:58,652 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.,
starting to roll back the global memstore size.
2013-10-31 00:07:58,653 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler:
Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.',
STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0',
ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK
2013-10-31 00:07:58,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070
Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to
RS_ZK_REGION_FAILED_OPEN
2013-10-31 00:07:58,670 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070
Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING
to RS_ZK_REGION_FAILED_OPEN
{noformat}

1) Initially AM try to assign region with 'bulk assign' on xxx100 (PENDING_OPEN => RS_ZK_REGION_OPENING);
but xxx100 failed to open region and AM handles this event (RS_ZK_REGION_FAILED_OPEN =>
CLOSED => OFFLINE)
2) AM try to assign region in ClosedRegionHandler on xxx108 (there is no RS_ZK_REGION_OPENING
event in master's logs, but we see it in regionserver's logs); it fails again
3) AM chose xxx106 for region assignment but receives RS_ZK_REGION_FAILED_OPEN before sending
request => CLOSED => ClosedRegionHandler => xxx104 => exception

> [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-8912
>                 URL: https://issues.apache.org/jira/browse/HBASE-8912
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>             Fix For: 0.94.14
>
>         Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html
>
>
> AM throws this exception which subsequently causes the master to abort: 
> {code}
> java.lang.IllegalStateException: Unexpected state : testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b.
state=PENDING_OPEN, ts=1372891751912, server=hemera.apache.org,39064,1372891746132 .. Cannot
transit it to OFFLINE.
> 	at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> 	at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> 	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> 	at java.lang.Thread.run(Thread.java:662)
> {code}
> This exception trace is from the failing test TestMetaReaderEditor which is failing pretty
frequently, but looking at the test code, I think this is not a test-only issue, but affects
the main code path. 
> https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message