From user-return-6428-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Fri Jun 14 20:55:15 2013 Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9631B10B7E for ; Fri, 14 Jun 2013 20:55:15 +0000 (UTC) Received: (qmail 95355 invoked by uid 500); 14 Jun 2013 20:55:15 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 95322 invoked by uid 500); 14 Jun 2013 20:55:14 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 95314 invoked by uid 99); 14 Jun 2013 20:55:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jun 2013 20:55:14 +0000 X-ASF-Spam-Status: No, hits=1.0 required=5.0 tests=FORGED_YAHOO_RCVD,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [212.82.109.200] (HELO nm24-vm9.bullet.mail.ird.yahoo.com) (212.82.109.200) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jun 2013 20:55:06 +0000 Received: from [77.238.189.48] by nm24.bullet.mail.ird.yahoo.com with NNFMP; 14 Jun 2013 20:54:46 -0000 Received: from [46.228.39.73] by tm1.bullet.mail.ird.yahoo.com with NNFMP; 14 Jun 2013 20:54:45 -0000 Received: from [127.0.0.1] by smtp110.mail.ir2.yahoo.com with NNFMP; 14 Jun 2013 20:54:45 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1371243285; bh=pQVZHLbt3CBFOef8HEvJMieYlAXuGgz5MqJSzCaeGIg=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:From:To:References:In-Reply-To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Mailer:Thread-Index:Content-Language; b=pyD5mCbKc58eL5xmg3bVFOiNQl8UuD5Q8NV8cLTni7ah/YdTIuUpCdhDX2y/ZIP9sod37U8x4UgxfaqEltcbEEO9Ho258piXedWGUQ9ohz7bxgIqqTYkfKB8itMwdhTcZqyJWz0g8488LZ/1bg+94HYIJWI+6WJ7YPaxjWieQQU= X-Yahoo-Newman-Id: 942543.88962.bm@smtp110.mail.ir2.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: ar1_qvIVM1l5lmddgsYFWFpKR.Q1BpAVyCIobVm7Im5CPFL Iafu2fNeHnCWSksl6O4zxVvSBtANeEXhkhVo4bF_u11OWpwP1Ruopcimfrxm W1zKA1Om1h7liKwUrlPdkAgDU22s9jjOpp8clW6YC3BvXgmQhC0ZwOvxP7sG 9ugGrq21Xo56iIt4bxGXeh235x5gfMiuQpQ0_MONVJGLmDj8sQcEwwmvVKRJ vY0CcfKEl07mWpegt0etjCSILOSpTWPBZwjcqp0c5lBVTEIdSYe3VDT5ys.Q dTFdn9QmOigFiJBtotXhq9Lw2V9e7lxoZtt6txq_mPK0dBKuguJ.l9d1czfW MD2686VBSTgKJa1TFnt0K19Ho70Frv8Qj40GbsOJN9kKLg4DSyMr9zw_gxRN gnKiKwIyzmNoTs.CPMtE.Qw33R08FShvDcBfJJIDtzKxoesRl9GpYxSNvnKE R3Qx4SoJKFllWs0JSiUtQJ1MW X-Yahoo-SMTP: HT5UJDeswBACWJPOeualxAa.da..S.fl X-Rocket-Received: from MSRCfpj (fpjunqueira@94.245.87.231 with ) by smtp110.mail.ir2.yahoo.com with SMTP; 14 Jun 2013 20:54:45 +0000 UTC From: "FPJ" To: References: <88D886D5-8D95-4C0F-A3F0-F252670D3B80@jordanzimmerman.com> <01ce01ce6932$7b922660$72b67320$@yahoo.com> <4EF64743-67B6-4B1D-B53D-942655C27F90@jordanzimmerman.com> <01d001ce6935$43a086e0$cae194a0$@yahoo.com> <575E4365-85ED-4451-A26B-08D1586D4AD6@jordanzimmerman.com> <01d201ce693f$e79d60b0$b6d82210$@yahoo.com> In-Reply-To: <01d201ce693f$e79d60b0$b6d82210$@yahoo.com> Subject: RE: Rolling config change considered harmful? Date: Fri, 14 Jun 2013 21:54:45 +0100 Message-ID: <01d401ce6941$66bf0af0$343d20d0$@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHFRSJNyo6mq+uOA1UsoJ3M6O/CKgL0qHD9AwrUroQB4ugQvQFY/vIuAeqDQJKY7sh8IA== Content-Language: en-gb X-Virus-Checked: Checked by ClamAV on apache.org I messed up the last sentence, here is what I was trying to say: It is ok to have two servers thinking they are leaders as long as only one is able to commit txns at a time by having a quorum of supporters. Each server is going to follow a single leader, so I don't see a problem in your scenario with the information you provided. Now if you tell me that when you keep sending new transactions to those leaders, both keep committing new transactions (not the same txns), then we have a problem. I don't see how this can happen, though. Also, one of the leaders should eventually time out and go back to leader election. > -----Original Message----- > From: FPJ [mailto:fpjunqueira@yahoo.com] > Sent: 14 June 2013 21:44 > To: user@zookeeper.apache.org > Subject: RE: Rolling config change considered harmful? > > It is ok to have two servers thinking they are leaders as long as only one is > able to commit txns at a time by having a quorum of supporters. Each server > is going to follow a single leader, so I don't see a problem in your scenario > with the information you provided. Now if you tell me that when you keep > sending new transactions to those leaders and they keep committing them > forever, both keep committing new transactions, then we have a problem. I > don't see how this can happen, though. > > Also, one of the leaders should eventually time out and go back to leader > election. > > -Flavio > > > -----Original Message----- > > From: Jordan Zimmerman [mailto:jordan@jordanzimmerman.com] > > Sent: 14 June 2013 21:10 > > To: user@zookeeper.apache.org > > Subject: Re: Rolling config change considered harmful? > > > > More on this. > > > > I just did some testing with wholly contrived scenarios and I was able > > to > get a > > cluster in a state where it had two leaders. NOTE: all of this was > > done > with > > Curator's TestingCluster > > > > * Create a 5 node ensemble > > * Save the list of instances, directories etc. > > * Wait for quorum > > * Shut down the cluster > > * Restart the ensemble with the same ports and directories. However, > > this time, give different server lists to each instance: > > * Instance A -> A D E > > * Instance B -> A B C > > * Instance C -> A B C > > * Instance D -> A D E > > * Instance E -> A D E > > > > There is at least one common server amongst all of them. When I > > restart > the > > cluster with this configuration I ended up with two leaders. This > > state > stays > > consistent after leader election (i.e. it doesn't try to re-elect). > > > > A: following > > B: leading > > C: following > > D: leading > > E: following > > > > This may be the correct behavior. i.e. it may be that ZooKeeper cannot > > realistically run in this scenario. What it means to me is that > > rolling > config > > changes, if too lax, can create chaos. > > > > -Jordan > > > > On Jun 14, 2013, at 12:27 PM, "FPJ" wrote: > > > > > In the case I described, the txn is not reflected in the zookeeper > state. > > > Say T is a create txn. Once C is elected, it determines the initial > > > history of txns for the new epoch that is starting and this initial > > > history is not going to include T. > > > > > > In the example below, I was ignoring the client that triggered T, > > > but since it has been acked by a quorum, the client might as well > > > have received the confirmation of the operation and think that the > > > znode has > > been created. > > > > > > -Flavio > > > > > >> -----Original Message----- > > >> From: Jordan Zimmerman [mailto:jordan@jordanzimmerman.com] > > >> Sent: 14 June 2013 20:16 > > >> To: user@zookeeper.apache.org > > >> Subject: Re: Rolling config change considered harmful? > > >> > > >> Yes - save that I'm not sure what happens with a client when a > > >> transaction > > > is > > >> lost. What is the error to the client? Or are you referring to > > >> internal transactions as part of the leader election? > > >> > > >> -JZ > > >> > > >> On Jun 14, 2013, at 12:07 PM, "FPJ" wrote: > > >> > > >>> Not sure if this helps but here is an example: > > >>> > > >>> - Txn T is acknowledged by A and B (ensemble is {A, B, C}) > > >>> - Ensemble changes to {B, C, D} > > >>> - C and D form a quorum and elect C because it has the highest zxid. > > >>> > > >>> C won't have T, so the txn gets lost. > > >>> > > >>> Does it make sense? > > >>> > > >>> -Flavio > > > > > > >