Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <1313993212.1256235299389.JavaMail.jira@brutus>
Date: Thu, 22 Oct 2009 18:14:59 +0000 (UTC)
From: "Yannis Pavlidis (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Updated: (HBASE-1928) ROOT and META tables stay in
 transition state (making the system not usable) if the designated
 regionServer dies before the assignment is complete
In-Reply-To: <1451204577.1256232779377.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yannis Pavlidis updated HBASE-1928:
-----------------------------------

    Attachment: diff_ServerManager.txt
                diff_RegionManager.txt
                diff_ProcessServerShutdown.txt

> ROOT and META tables stay in transition state (making the system not usable) if the designated regionServer dies before the assignment is complete
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1928
>                 URL: https://issues.apache.org/jira/browse/HBASE-1928
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.20.0, 0.20.1
>         Environment: Linux
>            Reporter: Yannis Pavlidis
>             Fix For: 0.20.2
>
>         Attachments: diff_ProcessServerShutdown.txt, diff_RegionManager.txt, diff_ServerManager.txt, master_cache01.txt, region_cache01.txt, region_cache02.txt
>
>
> During a ROOT or META table re-assignment if the designated regionServer dies before the assignment is complete then the whole cluster becomes unavailble since the ROOT or META tables cannot be accessed (and never recover since they are kept in a transition state).
> These are the 4 steps to replicate this issue (this is the easiest way to replicate. You can imagine that the following can occur in any real system).
> Pre condition
> ============
> 1. a cluster of 3 nodes (cache01, cache02, search01).
> 2. start the system (start-hbase)
> 3. cache02 has META, search01 has ROOT, cache01 has regionServer and Master.
> Case 1:
> =======
> 1. kill cache01
> 2. kill cache02
> 3. now search01 has both ROOT and META.
> 4. re-start RegionServers on cache01 and cache02
> 5. Tail the master logs and grep for "Assigning region -ROOT-" and also "Assigning region .META." (need to windows for easiness)
> 6. kill search01
> 7. wait to see to which server the ROOT will be assigned (from the tail)
> 8. quickly kill that server
> 9. you should notice that the ROOT server never gets re-assigned (because it is stuck in the regionsInTransitions)
> The termination occurs through the ServerManager::removeServerInfo since the regionServer sends back to the master in a report that it is shutting down.
> Case 2:
> ========
> Repeat Case1 and in step 7 and 8 kill the server that has the META region assigned to it. Again the cluster becomes unavailble because the META region stays in the regionsInTransitions.
> The termination occurs through the ServerManager::removeServerInfo since the regionServer sends back to the master in a report that it is shutting down.
> Case 3:
> ========
> Repeat Case1 and in step 7 and 8 kill the server with kill -9 instead of kill. This will not give the opportunity to the regionServer to send back the master in the report that it is terminating. The master will realize this because the znode will expire (but it is a different code path from before - it goes to the ProcessServerShutdown).
> Case 4:
> ========
> Repeat Case3 and in step 7 and 8 kill the server with kill -9 instead of kill. This will not give the opportunity to the regionServer to send back the master in the report that it is terminating. The master will realize this because the znode will expire (but it is a different code path from before - it goes to the ProcessServerShutdown).
> The solution would be to check the in the ServerManager:removeServerInfo and in  ProcessServerShutdown::closeMetaRegions whether the server that has been terminated has been assigned either the ROOT or META table. And if they have make sure we make those table ready to be re-assigned again.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.