zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Han (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (ZOOKEEPER-2678) Large databases take a long time to regain a quorum
Date Thu, 16 Feb 2017 00:08:41 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868839#comment-15868839
] 

Michael Han edited comment on ZOOKEEPER-2678 at 2/16/17 12:07 AM:
------------------------------------------------------------------

[~rakeshr] - yes it was deliberately skipped as I was not sure whether to put this in any
releasing branch or not (that was before the discussions on whether or not we put this in
3.4.10.). Now with this in 3.4.10 I don't see why it should not be included 3.5.3. So I will
merge this in 3.5. Thanks for the double check.

Now merged in branch-3.5
https://github.com/apache/zookeeper/commit/c164ee454cd70b3e65212ea1367ec86bb16ec022


was (Author: hanm):
[~rakeshr] - yes it was deliberately skipped as I was not sure whether to put this in any
releasing branch or not (that was before the discussions on whether or not we put this in
3.4.10.). Now with this in 3.4.10 I don't see why it should not be included 3.5.3. So I will
merge this in 3.5. Thanks for the double check.

> Large databases take a long time to regain a quorum
> ---------------------------------------------------
>
>                 Key: ZOOKEEPER-2678
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2678
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.4.9, 3.5.2
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 3.4.10, 3.5.3, 3.6.0
>
>
> I know this is long but please here me out.
> I recently inherited a massive zookeeper ensemble.  The snapshot is 3.4 GB on disk. 
Because of its massive size we have been running into a number of issues. There are lots of
problems that we hope to fix with tuning GC etc, but the big one right now that is blocking
us making a lot of progress on the rest of them is that when we lose a quorum because the
leader left, for what ever reason, it can take well over 5 mins for a new quorum to be established.
 So we cannot tune the leader without risking downtime.
> We traced down where the time was being spent and found that each server was clearing
the database so it would be read back in again before leader election even started.  Then
as part of the sync phase each server will write out a snapshot to checkpoint the progress
it made as part of the sync.
> I will be putting up a patch shortly with some proposed changes in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message