hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4792) SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region stuck offline
Date Wed, 16 Nov 2011 06:10:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151047#comment-13151047
] 

Hudson commented on HBASE-4792:
-------------------------------

Integrated in HBase-TRUNK #2445 (See [https://builds.apache.org/job/HBase-TRUNK/2445/])
    Adding the 0.92 entry for HBASE-4792
HBASE-4792  SplitRegionHandler doesn't care if it deletes the znode or not,
               leaves the parent region stuck offline

jdcryans : 
Files : 
* /hbase/trunk/CHANGES.txt

jdcryans : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/SplitRegionHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java

                
> SplitRegionHandler doesn't care if it deletes the znode or not, leaves the parent region
stuck offline
> ------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4792
>                 URL: https://issues.apache.org/jira/browse/HBASE-4792
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.92.0
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>            Priority: Critical
>             Fix For: 0.92.0, 0.94.0
>
>         Attachments: HBASE-4792-0.92.patch
>
>
> Saw this on a little test cluster, really easy to trigger.
> First the master log:
> {quote}
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.master.handler.SplitRegionHandler:
Handling SPLIT event for e5be6551c8584a6a1065466e520faf4e; deleting node
> 2011-11-15 22:28:57,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:62003-0x132f043bbde08c1
Deleting existing unassigned node for e5be6551c8584a6a1065466e520faf4e that is in expected
state RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,975 WARN org.apache.hadoop.hbase.zookeeper.ZKAssign: master:62003-0x132f043bbde08c1
Attempting to delete unassigned node in RS_ZK_REGION_SPLIT state but after verifying state,
we got a version mismatch
> 2011-11-15 22:28:57,975 INFO org.apache.hadoop.hbase.master.handler.SplitRegionHandler:
Handled SPLIT report); parent=TestTable,0001355346,1321396080924.e5be6551c8584a6a1065466e520faf4e.
daughter a=TestTable,0001355346,1321396132414.df9b549eb594a1f8728608a2a431224a.daughter b=TestTable,0001368082,1321396132414.de861596db4337dc341138f26b9c8bc2.
> ...
> 2011-11-15 22:28:58,052 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_SPLIT, server=sv4r28s44,62023,1321395865619, region=e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:58,052 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region
e5be6551c8584a6a1065466e520faf4e not found on server sv4r28s44,62023,1321395865619; failed
processing
> 2011-11-15 22:28:58,052 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received
SPLIT for region e5be6551c8584a6a1065466e520faf4e from server sv4r28s44,62023,1321395865619
but it doesn't exist anymore, probably already processed its split
> (repeated forever)
> {quote}
> The master processes the split but when it calls ZKAssign.deleteNode it doesn't check
the boolean that's returned. In this case it was false. So for the master the split was completed,
but for the region server it's another story:
> {quote}
> 2011-11-15 22:28:57,661 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Attempting to transition node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING
to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Successfully transitioned node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLITTING
to RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,775 INFO org.apache.hadoop.hbase.regionserver.SplitTransaction: Still
waiting on the master to process the split for e5be6551c8584a6a1065466e520faf4e
> 2011-11-15 22:28:57,876 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Attempting to transition node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to
RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:57,967 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Successfully transitioned node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to
RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Attempting to transition node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to
RS_ZK_REGION_SPLIT
> 2011-11-15 22:28:58,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:62023-0x132f043bbde08d3
Successfully transitioned node e5be6551c8584a6a1065466e520faf4e from RS_ZK_REGION_SPLIT to
RS_ZK_REGION_SPLIT
> (printed forever)
> {quote}
> Since the znode isn't really deleted, it thinks the master just haven't got to process
its region thus waits which leaves the region *unavailable*.
> We need to just retry the delete master-side ASAP since the RS will wait 100ms between
retries.
> At the same time, it would be nice if ZKAssign.deleteNode always printed out the name
of the region in its messages because it took me a while to see that the delete didn't take
affect while looking at a grep.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message