hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yannis Pavlidis (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1928) ROOT and META tables stay in transition state (making the system not usable) if the designated regionServer dies before the assignment is complete
Date Thu, 22 Oct 2009 18:16:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768799#action_12768799
] 

Yannis Pavlidis commented on HBASE-1928:
----------------------------------------

stack.

I am re-assigning this too for further testing and approval. 

I performed all the tests described in the first comments without the patches to replicate
the issue and then performed the same steps with the patches and everything worked as expected
(at the end ROOT and META were assigned to a regionServer).

Thanks in advance,

Yannis.

> ROOT and META tables stay in transition state (making the system not usable) if the designated
regionServer dies before the assignment is complete
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1928
>                 URL: https://issues.apache.org/jira/browse/HBASE-1928
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.20.0, 0.20.1
>         Environment: Linux
>            Reporter: Yannis Pavlidis
>             Fix For: 0.20.2
>
>         Attachments: diff_ProcessServerShutdown.txt, diff_RegionManager.txt, diff_ServerManager.txt,
master_cache01.txt, region_cache01.txt, region_cache02.txt
>
>
> During a ROOT or META table re-assignment if the designated regionServer dies before
the assignment is complete then the whole cluster becomes unavailble since the ROOT or META
tables cannot be accessed (and never recover since they are kept in a transition state).
> These are the 4 steps to replicate this issue (this is the easiest way to replicate.
You can imagine that the following can occur in any real system).
> Pre condition
> ============
> 1. a cluster of 3 nodes (cache01, cache02, search01).
> 2. start the system (start-hbase)
> 3. cache02 has META, search01 has ROOT, cache01 has regionServer and Master.
> Case 1:
> =======
> 1. kill cache01
> 2. kill cache02
> 3. now search01 has both ROOT and META.
> 4. re-start RegionServers on cache01 and cache02
> 5. Tail the master logs and grep for "Assigning region -ROOT-" and also "Assigning region
.META." (need to windows for easiness)
> 6. kill search01
> 7. wait to see to which server the ROOT will be assigned (from the tail)
> 8. quickly kill that server
> 9. you should notice that the ROOT server never gets re-assigned (because it is stuck
in the regionsInTransitions)
> The termination occurs through the ServerManager::removeServerInfo since the regionServer
sends back to the master in a report that it is shutting down.
> Case 2:
> ========
> Repeat Case1 and in step 7 and 8 kill the server that has the META region assigned to
it. Again the cluster becomes unavailble because the META region stays in the regionsInTransitions.
> The termination occurs through the ServerManager::removeServerInfo since the regionServer
sends back to the master in a report that it is shutting down.
> Case 3:
> ========
> Repeat Case1 and in step 7 and 8 kill the server with kill -9 instead of kill. This will
not give the opportunity to the regionServer to send back the master in the report that it
is terminating. The master will realize this because the znode will expire (but it is a different
code path from before - it goes to the ProcessServerShutdown).
> Case 4:
> ========
> Repeat Case3 and in step 7 and 8 kill the server with kill -9 instead of kill. This will
not give the opportunity to the regionServer to send back the master in the report that it
is terminating. The master will realize this because the znode will expire (but it is a different
code path from before - it goes to the ProcessServerShutdown).
> The solution would be to check the in the ServerManager:removeServerInfo and in  ProcessServerShutdown::closeMetaRegions
whether the server that has been terminated has been assigned either the ROOT or META table.
And if they have make sure we make those table ready to be re-assigned again.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message