Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 10 Feb 2017 02:29:41 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13025901.1481017107000.44699.1486693781959@Atlassian.JIRA>
In-Reply-To: <JIRA.13025901.1481017107000@Atlassian.JIRA>
References: <JIRA.13025901.1481017107000@Atlassian.JIRA> <JIRA.13025901.1481017107544@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17265) Region left unassigned in master
 failover when region failed to open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 10 Feb 2017 02:29:49 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860567#comment-15860567 ] 

Hudson commented on HBASE-17265:
--------------------------------

FAILURE: Integrated in Jenkins build HBase-1.4 #622 (See [https://builds.apache.org/job/HBase-1.4/622/])
HBASE-17265 Region left unassigned in master failover when region failed (tedyu: rev 52c51fc271b8bf0e3dfc8dc05513cc61ce689787)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java


> Region left unassigned in master failover when region failed to open
> --------------------------------------------------------------------
>
>                 Key: HBASE-17265
>                 URL: https://issues.apache.org/jira/browse/HBASE-17265
>             Project: HBase
>          Issue Type: Bug
>          Components: Region Assignment
>    Affects Versions: 1.1.7
>            Reporter: Allan Yang
>            Assignee: Allan Yang
>             Fix For: 1.4.0, 1.3.1
>
>         Attachments: HBASE-17265-branch-1.patch, HBASE-17265-branch-1.v2.patch
>
>
> This problem is very similar with HBASE-13330. It is also a result of ServerShutdownHandler and AssignmentManager 'thought' the region will be assigned by each other, and left the region remain unassigned.
> But HBASE-13330 only dealed with RS_ZK_REGION_FAILED_OPEN in {{processRegionInTransition}}.  
> Region failed open may happen after {{processRegionInTransition}}. In my case, when master failover, it assigned all RIT regions, but some are failed to open(due to HBASE-17264), AssignmentManager received the zk event, and skip to assign it(this region was opened on a failed server before and already in RIT before master failover). The SSH also skip to assign it because it was RIT on another RS.
> Master recevied a zk event of RS_ZK_REGION_FAILED_OPEN and begin to handle it:
> {noformat}
> 2016-11-23 17:11:46,944 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, server=example.org,30003,1479780976834, region=57513956a7b671f4e8da1598c2e2970e, current_state={57513956a7b671f4e8da1598c2e2970e state=PENDING_OPEN, ts=1479892306843, server=example.org,30003,1479780976834}
> 2016-11-23 17:11:46,944 INFO  [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {57513956a7b671f4e8da1598c2e2970e state=PENDING_OPEN, ts=1479892306843, server=example.org,30003,1479780976834} to {57513956a7b671f4e8da1598c2e2970e state=CLOSED, ts=1479892306944, server=example.org,30003,1479780976834}
> 2016-11-23 17:11:46,945 WARN  [AM.ZK.Worker-pool2-t1] master.RegionStates: 57513956a7b671f4e8da1598c2e2970e moved to CLOSED on example.org,30003,1479780976834, expected example.org,30003,1475893095003
> 2016-11-23 17:11:46,950 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager: Found an existing plan for test,QFO7M,1475986053104.57513956a7b671f4e8da1598c2e2970e. destination server is example.org,30003,1479780976834 accepted as a dest server = false
> 2016-11-23 17:11:47,012 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for test,QFO7M,1475986053104.57513956a7b671f4e8da1598c2e2970e.; generated random plan=hri=test,QFO7M,1475986053104.57513956a7b671f4e8da1598c2e2970e., src=, dest=11.239.21.235,30003,1479781410131; 2 (online=3) available servers, forceNewPlan=true
> 2016-11-23 17:11:47,014 DEBUG [AM.ZK.Worker-pool2-t1] handler.ClosedRegionHandler: Handling CLOSED event for 57513956a7b671f4e8da1598c2e2970e
> 2016-11-23 17:11:47,015 WARN  [AM.ZK.Worker-pool2-t1] master.RegionStates: 57513956a7b671f4e8da1598c2e2970e moved to CLOSED on example.org,30003,1479780976834, expected example.org,30003,1475893095003
> {noformat}
> AssignmentManager skip to assign it because the region was on a failed server
> {noformat}
> 2016-11-23 17:11:47,017 INFO  [AM.ZK.Worker-pool2-t1] master.AssignmentManager: Skip assigning test,QFO7M,1475986053104.57513956a7b671f4e8da1598c2e2970e., it's host example.org,30003,1475893095003 is dead but not processed yet
> {noformat}
> SSH also skip it because it was RIT on another server
> {noformat}
> 2016-11-23 17:12:17,850 INFO  [MASTER_SERVER_OPERATIONS-example.org:30001-0] master.RegionStates: Transitioning {57513956a7b671f4e8da1598c2e2970e state=CLOSED, ts=1479892307015, server=example.org,30003,1479780976834} will be handled by SSH for example.org,30003,1475893095003
> 2016-11-23 17:12:17,910 INFO  [MASTER_SERVER_OPERATIONS-example.org:30001-0] handler.ServerShutdownHandler: Skip assigning region in transition on other server{57513956a7b671f4e8da1598c2e2970e state=CLOSED, ts=1479892307015, server=example.org,30003,1479780976834}
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)