hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5270) Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler
Date Wed, 08 Feb 2012 20:08:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203921#comment-13203921

stack commented on HBASE-5270:

I was taking a look through HBASE-5179 and HBASE-4748 again, the two issues that spawned this
one (Both are in synopsis about master failover with concurrent servershutdown handler running).
 I have also been looking at "HBASE-5344
[89-fb] Scan unassigned region directory on master failover".

HBASE-5179 starts out as we can miss edits if a server is discovered to be dead AFTER master
failover has started up splitting logs because we'll notice it dead so will assign out its
regions but before we've had a chance to split its logs.  The way fb deal with this in hbase-5344
is not to process zookeeper events that come in during master failover.  They queue them instead
and only start in on the processing after master is up.

Chunhui does something like this in his original patch by adding any server currently being
processed by server shutdown to the list of regionservers whose logs we should not split.
 The fb way of halting temporarily the callback processing seems more airtight.

HBASE-5179 is then extended to include as in scope, the processing of servers carrying root
and meta (hbase-4748) that crash during master failover.  We need to consider the cases where
a server crashes AFTER master failover distributed log splitting has started but before we
run the verifications of meta and root locations.

Currently we'll expire the server that is unresponsive when we go to verify root and meta
locations.  The notion is that the meta regions will be assigned by the server shutdown handler.
 The fb technique of turning off processing zk events would mess with our existing handling
code here -- but I'm not too confident the code is going to do the right thing since it has
no tests of this predicament and the scenarios look like they could be pretty varied (root
is offline only, meta server has crashed only, a server with both root and meta has crashed,
etc).  In hbase-5344, fb will go query each regionserver for the regions its currently hosting
(and look in zk to see what rs are up).   Maybe we need some of this from 89-fb in trunk but
I'm not clear on it just yet; would need more study of the current state of trunk and then
of what is happening over in 89-fb.

One thing I think we should do to lessen the number of code paths we can take on failover
is to do the long-talked of purge  of the root region.  This should cut down on the number
of states we need to deal with and make reasoning about failure states on failover easier
to reason about.
> Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler
> -----------------------------------------------------------------------------------------------------
>                 Key: HBASE-5270
>                 URL: https://issues.apache.org/jira/browse/HBASE-5270
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master
>            Reporter: Zhihong Yu
>             Fix For: 0.94.0, 0.92.1
> This JIRA continues the effort from HBASE-5179. Starting with Stack's comments about
patches for 0.92 and TRUNK:
> Reviewing 0.92v17
> isDeadServerInProgress is a new public method in ServerManager but it does not seem to
be used anywhere.
> Does isDeadRootServerInProgress need to be public? Ditto for meta version.
> This method param names are not right 'definitiveRootServer'; what is meant by definitive?
Do they need this qualifier?
> Is there anything in place to stop us expiring a server twice if its carrying root and
> What is difference between asking assignment manager isCarryingRoot and this variable
that is passed in? Should be doc'd at least. Ditto for meta.
> I think I've asked for this a few times - onlineServers needs to be explained... either
in javadoc or in comment. This is the param passed into joinCluster. How does it arise? I
think I know but am unsure. God love the poor noob that comes awandering this code trying
to make sense of it all.
> It looks like we get the list by trawling zk for regionserver znodes that have not checked
in. Don't we do this operation earlier in master setup? Are we doing it again here?
> Though distributed split log is configured, we will do in master single process splitting
under some conditions with this patch. Its not explained in code why we would do this. Why
do we think master log splitting 'high priority' when it could very well be slower. Should
we only go this route if distributed splitting is not going on. Do we know if concurrent distributed
log splitting and master splitting works?
> Why would we have dead servers in progress here in master startup? Because a servershutdownhandler
> This patch is different to the patch for 0.90. Should go into trunk first with tests,
then 0.92. Should it be in this issue? This issue is really hard to follow now. Maybe this
issue is for 0.90.x and new issue for more work on this trunk patch?
> This patch needs to have the v18 differences applied.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message