hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5094) The META can hold an entry for a region with a different server name from the one actually in the AssignmentManager thus making the region inaccessible.
Date Mon, 26 Dec 2011 06:50:30 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175899#comment-13175899

ramkrishna.s.vasudevan commented on HBASE-5094:

RegionState rit = this.services.getAssignmentManager().isRegionInTransition(e.getKey());
            ServerName addressFromAM = this.services.getAssignmentManager()
            if (rit != null && !rit.isClosing() && !rit.isPendingClose())
              // Skip regions that were in transition unless CLOSING or
              // PENDING_CLOSE
              LOG.info("Skip assigning region " + rit.toString());
            } else if (addressFromAM != null
                && !addressFromAM.equals(this.serverName)) {
              LOG.debug("Skip assigning region "
                    + e.getKey().getRegionNameAsString()
                    + " because it has been opened in "
                    + addressFromAM.getServerName());
In ServerShutDownHandler we try to get the address in the AM.  This address is initially null
because it is not yet updated after the region was opened .i.e. the CAll back after node deletion
is not yet done in the master side.
But removal from RIT is completed on the master side.  So this will trigger a new assignment.
So there is a small window between the online region is actually added in to the online list
and the ServerShutdownHandler where we check the existing address in AM.
> The META can hold an entry for a region with a different server name from the one actually
in the AssignmentManager thus making the region inaccessible.
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HBASE-5094
>                 URL: https://issues.apache.org/jira/browse/HBASE-5094
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
> R1 is reassigned to RS3 during RS1 shutdown, even though R1 was just assigned to RS2
by load balancer. So .META. table indicated R1 is on RS3. Both RS2 and RS3 think they have
R1. Later when RS3 shutdown, R1 is reassigned to RS2. RS2 will indicate ALREADY_OPENED. Thus
the region is considered assigned to RS2 even though .META. indicates it is on RS3.
> T1: Load balancer tried to move R1 from RS1 to RS2
> . 2011-11-21 14:03:20,812 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.,
src=skynet-1,60020,1321912978281, dest=skynet-4,60020,1321912999305
> T2: RS1 shutdown. 2011-11-21 14:03:24,759 DEBUG org.apache.hadoop.hbase.master.ServerManager:
Added=skynet-1,60020,1321912978281 to dead servers, submitted shutdown handler to be executed,
root=false, meta=true
> T3:R1 is opened on RS2. 2011-11-21 14:03:26,131 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler:
The master has opened the region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.
that was online on skynet-4,60020,1321912999305
> T4: As part of RS1 shutdown handling, region reassignment starts. It uses the region
location captured at T2. 2011-11-21 14:03:26,152 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Reassigning 32 region(s) that skynet-1,60020,1321912978281 was carrying (skipping 0 regions(s)
that are already in transition)
> T5: R1 is assigned to RS3. 2011-11-21 14:03:27,404 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil:
master:60000-0x133b84f9f490000 Retrieved 115 byte(s) of data from znode /hbase/unassigned/ee2e205a60f1bb06cc73bc9df06289df;
origin=skynet-3,60020,1321912991430, state=RS_ZK_REGION_OPENED
> T6: RS3 shutdown. R1 is reassigned to RS2. 2011-11-21 14:03:37,899 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
ALREADY_OPENED region tableXY,\xB8Q\xEB\x85\x1E\xB8Q\xDF,1321573099841.ee2e205a60f1bb06cc73bc9df06289df.
to skynet-4,60020,1321912999305
> From AssignmentManager point of view, the R1 is assigned to RS2. The .META. table indicates
the location is RS3.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message