From user-return-4961-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Thu May 10 08:29:40 2012 Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 103FF9A9B for ; Thu, 10 May 2012 08:29:40 +0000 (UTC) Received: (qmail 32887 invoked by uid 500); 10 May 2012 08:29:39 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 32535 invoked by uid 500); 10 May 2012 08:29:37 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 32508 invoked by uid 99); 10 May 2012 08:29:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 May 2012 08:29:36 +0000 X-ASF-Spam-Status: No, hits=-3.7 required=5.0 tests=RCVD_IN_DNSWL_HI,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Alexandar.Gvozdenovic@ubs.com designates 139.149.1.205 as permitted sender) Received: from [139.149.1.205] (HELO dmz-vsgate2.ldn.ibb.ubs.com) (139.149.1.205) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 May 2012 08:29:27 +0000 Received: from localhost (localhost [127.0.0.1]) by postfix.amavisd (Postfix) with ESMTP id 479348C72D for ; Thu, 10 May 2012 09:28:51 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ubs.com; h= x-virus-scanned:from:references:in-reply-to:message-id:date :subject:content-transfer-encoding:content-type:mime-version :content-class:received:received:received:received:received; s= srsa; t=1336638531; bh=lGmj9kp+A8DT0LAdDaikRKHBAn+/JrS3/AvRsqxFT Sk=; b=h16SEufHpPCm9BVAS3j56ACXRxb5/Pp3zlqByRx9JzyYgG9rbVWu/wiP2 NsQx62yr44sITPXEvjLaCWqsiIEbBvozUGgrDkiJ4dJhEx2YLbCK1SFBAwbjlTtp SPcx5n7DNT1O1FWAzxaR46Fj6lDVcDMLO0BHCEUS/G2mMWqbg4= Received: from dmz-vsgate2.ldn.ibb.ubs.com ([127.0.0.1]) by localhost (sldn0848xmh.ldn.swissbank.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G10LtgWBFyCm for ; Thu, 10 May 2012 09:28:51 +0100 (BST) Received: by dmz-vsgate2.ldn.ibb.ubs.com (Postfix, from userid 103) id 1B5098C460; Thu, 10 May 2012 09:28:46 +0100 (BST) Received: from sldn3028pmh.ldn.swissbank.com (unknown [139.149.96.13]) by dmz-vsgate2.ldn.ibb.ubs.com (Postfix) with ESMTP id B85F98C603 for ; Thu, 10 May 2012 09:28:45 +0100 (BST) Received: from nldn2376pap.ubsw.net (nldn2376pap.ldn.swissbank.com [14.64.49.25]) by sldn3028pmh.ldn.swissbank.com (Postfix) with ESMTP id 97E5C35F for ; Thu, 10 May 2012 09:28:45 +0100 (BST) Received: from NLDNC108PEX1.ubsw.net ([139.149.84.162]) by nldn2376pap.ubsw.net with Microsoft SMTPSVC(6.0.3790.4675); Thu, 10 May 2012 09:28:45 +0100 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: Possible issue with cluster availability following new Leader Election - ZK 3.4 Date: Thu, 10 May 2012 09:28:44 +0100 Message-ID: <26D6F1977403764E8B5C4B26F9430E0A0FA2E4C2@NLDNC108PEX1.ubsw.net> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Possible issue with cluster availability following new Leader Election - ZK 3.4 Thread-Index: Ac0uB8BQVl2nwe0jRLiOv/Prj2fccQAfo4RQ References: <26D6F1977403764E8B5C4B26F9430E0A0FA2E277@NLDNC108PEX1.ubsw.net> From: To: X-OriginalArrivalTime: 10 May 2012 08:28:45.0360 (UTC) FILETIME=[EA374B00:01CD2E86] X-Virus-Scanned: clamav-milter 0.97.3 at sldn3028pmh.ldn.swissbank.com X-Virus-Status: Clean X-UBS-Disclaimer: Version $Revision: 1.33 (postfix edition)$ Looks like it is pretty closely correlated to data set size. With an empty set, it takes a couple of seconds (still quite high?). With a set half the size it takes roughly half the time.=20 I had issues where it would never settle down once the set got above a certain size. I worked around this by increasing the 'initLimit' and the 'syncLimit' properties. The latter controlled how long the leader would wait for the follower to ACK his snapshot push.=20 =20 -----Original Message----- From: Mark Gius [mailto:mgius7096@gmail.com]=20 Sent: 09 May 2012 18:18 To: user@zookeeper.apache.org Subject: Re: Possible issue with cluster availability following new Leader Election - ZK 3.4 I'm encountering a similar issue with a more or less empty dataset. I bring up a cluster of 3 servers and shoot one of them in the head. It takes ~20 seconds for the two remaining hosts to settle and begin responding the requests again. If you're certain that your delay is due to pushing down the larger dataset then we may be seeing different problems. Mark On Wed, May 9, 2012 at 6:27 AM, wrote: > > Hi Zookeeper devs and users, > > > I've been doing some load and failover testing on the ZK 3.4 branch=20 > using moderately large data sets (700mb and 20k nodes) and I think=20 > there could be an issue. > > When I bring down the leader of a 3 node cluster, it takes around=20 > 20-30 seconds for the cluster as a whole to become available again. > This is because once a new leader is elected it pushes out a snapshot=20 > to all the peers who in turn persist it locally before sending an ack back. > Only then does the leader decide he has a valid quorum. In this case=20 > pretty much all the time is taken up sending the data over the network > and re-saving it. > > Granted I'm testing this on some low-spec VM's so I wouldn't expect a=20 > real-world sync for a data set that size to take anything like as long. > However is this not a significant constraint on availability if,=20 > whenever a leader fails, a full snapshot needs to sent to and=20 > persisted by a quorum of peers before the cluster as a whole can be=20 > deemed as available? > > I notice when a peer joins a stable cluster as a follower,=20 > synchronization is implemented via diffs and the peer is quickly=20 > available for client connections provided it already had an up to date > local state. > Should not something similar not be possible when a new leader is=20 > elected. A quick glance at the code (line 390 of LearnerHandler)=20 > suggests there is some logic to send an empty diff but I never see=20 > this triggered. > > I'm am not mutating any state in the cluster whilst I am bringing=20 > stuff up and down so is this behaviour a bug or by design? > > I saw a related question > (http://zookeeper-user.578899.n2.nabble.com/leader-election-length-td7 > 08 > 6868.html#a7089472) a few months back that touched on this, but there=20 > was not much follow up. > > Many thanks > > Alex > > > > > > Visit our website at http://www.ubs.com > > This message contains confidential information and is intended only=20 > for the individual named. If you are not the named addressee you=20 > should not disseminate, distribute or copy this e-mail. Please notify=20 > the sender immediately by e-mail if you have received this e-mail by=20 > mistake and delete this e-mail from your system. > > E-mails are not encrypted and cannot be guaranteed to be secure or=20 > error-free as information could be intercepted, corrupted, lost,=20 > destroyed, arrive late or incomplete, or contain viruses. The sender=20 > therefore does not accept liability for any errors or omissions in the > contents of this message which arise as a result of e-mail transmission. > If verification is required please request a hard-copy version. This=20 > message is provided for informational purposes and should not be=20 > construed as a solicitation or offer to buy or sell any securities or=20 > related financial instruments. > > UBS Limited is a company limited by shares incorporated in the United=20 > Kingdom registered in England and Wales with number 2035362. > Registered office: 1 Finsbury Avenue, London EC2M 2PP. UBS Limited is > authorised and regulated by the Financial Services Authority. > > UBS AG is a public company incorporated with limited liability in=20 > Switzerland domiciled in the Canton of Basel-City and the Canton of=20 > Zurich respectively registered at the Commercial Registry offices in=20 > those Cantons with Identification No: CH-270.3.004.646-4 and having=20 > respective head offices at Aeschenvorstadt 1, 4051 Basel and=20 > Bahnhofstrasse 45, 8001 Zurich, Switzerland. Registered in the United > Kingdom as a foreign company with No: FC021146 and having a UK=20 > Establishment registered at Companies House, Cardiff, with No: > BR 004507. The principal office of UK Establishment: 1 Finsbury=20 > Avenue, London EC2M 2PP. In the United Kingdom, UBS AG is authorised=20 > and regulated by the Financial Services Authority. > > UBS reserves the right to retain all messages. Messages are protected=20 > and accessed only in legally justified cases. > Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mails are not encrypted and cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. UBS Limited is a company limited by shares incorporated in the United Kingdom registered in England and Wales with number 2035362. Registered office: 1 Finsbury Avenue, London EC2M 2PP. UBS Limited is authorised and regulated by the Financial Services Authority. UBS AG is a public company incorporated with limited liability in Switzerland domiciled in the Canton of Basel-City and the Canton of Zurich respectively registered at the Commercial Registry offices in those Cantons with Identification No: CH-270.3.004.646-4 and having respective head offices at Aeschenvorstadt 1, 4051 Basel and Bahnhofstrasse 45, 8001 Zurich, Switzerland. Registered in the United Kingdom as a foreign company with No: FC021146 and having a UK Establishment registered at Companies House, Cardiff, with No: BR 004507. The principal office of UK Establishment: 1 Finsbury Avenue, London EC2M 2PP. In the United Kingdom, UBS AG is authorised and regulated by the Financial Services Authority. UBS reserves the right to retain all messages. Messages are protected and accessed only in legally justified cases.