From user-return-12855-apmail-zookeeper-user-archive=zookeeper.apache.org@zookeeper.apache.org Thu Jun 18 08:54:52 2020 Return-Path: X-Original-To: apmail-zookeeper-user-archive@www.apache.org Delivered-To: apmail-zookeeper-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with SMTP id AD8C819E86 for ; Thu, 18 Jun 2020 08:54:50 +0000 (UTC) Received: (qmail 60869 invoked by uid 500); 18 Jun 2020 08:54:47 -0000 Delivered-To: apmail-zookeeper-user-archive@zookeeper.apache.org Received: (qmail 60831 invoked by uid 500); 18 Jun 2020 08:54:47 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 60818 invoked by uid 99); 18 Jun 2020 08:54:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2020 08:54:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1F81A181327 for ; Thu, 18 Jun 2020 08:54:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id BWgoBD8T6XZL for ; Thu, 18 Jun 2020 08:54:42 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.53; helo=mail-ed1-f53.google.com; envelope-from=szalay.beko.mate@gmail.com; receiver= Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id E7F25BB802 for ; Thu, 18 Jun 2020 08:54:41 +0000 (UTC) Received: by mail-ed1-f53.google.com with SMTP id t21so4202360edr.12 for ; Thu, 18 Jun 2020 01:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=RzkybZzWSc4ddK4Th9Iybhh3LbgP31X37zqLgy9mTP8=; b=f4sSOt2MigWcBQNbJqKom0j8wPi22Ty1EmldBobOINvXaj1Znc9j+oy5XqiquHQa88 WO5SUrZZ57gEYXwhm85F1VMhr3+7PCEPmaGXd8kp1WHuByLjCAfFJ2fjX+aQ6+TMaUbX ljuA8jVipCk98GW651luNgpBjW9U7VKBhNHMkVJIEcEDMr9II5BWXqcttyQgL7f7qRET KojZLTK8QlglB2NYIlpHKzbFWFrWwuNydSPAg+M2hzEDA++JGvjF4omuKCLhpGSyxxFh XhD6d0ceWy/4enFJ7SuzVjxsLVVhpr20x89guuvp+GIn73ilW7TanC7TqS8Am/loMc3+ PX+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=RzkybZzWSc4ddK4Th9Iybhh3LbgP31X37zqLgy9mTP8=; b=SbTgOuu+2Gttk7+8CLvIvutKnnnH5hUu6AQdv0Ex8ZbC17Ovfz7i2O1cP4+0GEbVRE umj+fsAHesWJ+WTL+W4hv/ezD3NECIBsB0p9miW1JetlcGBwtzvq/g8mX/t+skANtXYl Kv1NpoJXyZUejsnMCC2ujPyBzGSu6roPTGNYJFFJR7Kl+G8iwRm3Hx3JjSIBTyoGLRiY 0hnR8MEKfLfcfGqtNZp0TQs3lYIXIiEkZNyrXXcpWjzV9ke1GOaUq3RlVmY9txrcy39E wt6Bg2z5Va6TkqFgzZAwFUmSvgPKM4ZC/5NanBiIGC6w0+oVyJhzJj9sxa+dnZUt+RgX vcWA== X-Gm-Message-State: AOAM532Y0oape8TmhyGmvepZt+NuYTgaoFx3IzxDRgmNIQ/ADdYcFkhL UIL/8LS8E6CBMKBaOL9vsh2TBFw1+Zo7NLjZl+OCS4aumw== X-Google-Smtp-Source: ABdhPJysdgjcI5si6Rn45VOmnIiHKQPlj0D1bpRteSf5QrUv2r5Znya5pdAaabiekWRgNwZu8vxcPqXFzEF9paJ5MKY= X-Received: by 2002:a05:6402:1d96:: with SMTP id dk22mr3189759edb.258.1592470480838; Thu, 18 Jun 2020 01:54:40 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?U3phbGF5LUJla8WRIE3DoXTDqQ==?= Date: Thu, 18 Jun 2020 10:54:29 +0200 Message-ID: Subject: Re: How to deliberately cause a split brain? To: UserZooKeeper Content-Type: multipart/alternative; boundary="00000000000053655705a857eda0" --00000000000053655705a857eda0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Even with using different server configs, no vote from "unknown" server should be accepted by the others. (at least if dynamic reconfig is disabled) However, I never really tested with intentionally bad configs. There might exist some bugs we are unaware of and haven't fixed yet. There was an important bugfix recently which hasn't been released yet: https://github.com/apache/zookeeper/pull/1356 - it is always possible that we still have holes somewhere. But still, I think any such scenario that leads to a split-brain should be considered as a bug. On Thu, Jun 18, 2020 at 10:43 AM Szalay-Bek=C5=91 M=C3=A1t=C3=A9 < szalay.beko.mate@gmail.com> wrote: > > You can just use iptables and simulate that some network paths are not > working properly > I would assume ZooKeeper would handle this, and if no partition has > quorum, then no writes will be enabled / no leader will be present. > (readonly mode can still work, if configured properly) > > Honestly, I can not really think of any way to get split-brain. If someon= e > can, then let us know so that we can fix it ;) > > Cheers, > Mate > > On Thu, Jun 18, 2020 at 10:34 AM Enrico Olivelli > wrote: > >> Tim >> You can just use iptables and simulate that some network paths are not >> working properly >> >> Enrico >> >> Il Gio 18 Giu 2020, 10:26 Tim Ward ha >> scritto: >> >> > I have been tasked with writing some monitoring code to detect a >> Zookeeper >> > split brain using the monitoring system. >> > >> > >> > >> > All well and good, I think I can see how to do that, but what about >> > testing? How can I deliberately provoke a Zookeeper ensemble into goin= g >> > into a split brain state so that I can test the detection code? >> > >> > >> > >> > Scenarios might be along the lines of: the ops people want to increase >> an >> > ensemble size, but get something wrong in the (manual) writing of the >> > configuration files or in the (manual) restarting of instances. What >> > "something" is likely to work here? >> > >> > >> > >> > >> > >> > *Tim Ward* >> > >> > >> > >> > >> > >> > Principal engineer >> > >> > >> > >> > >> > >> > =E2=80=94 >> > >> > >> > >> > >> > >> > *t:=E2=80=AF*+44 (0) 1223 345 940 >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > *Broers Building, Hauser Forum* >> > >> > *21 JJ Thomson Avenue, Cambridge* >> > >> > *CB3 0FA United Kingdom* >> > >> > >> > >> > >> > >> > featurespace.com *=E2=80=AF|=E2=80=AF*T= witter >> > *=E2=80=AF|=E2=80=AF*LinkedIn >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > [image: A screenshot of a cell phone Description automatically >> generated] >> > >> > >> > >> > >> > This message, and any files/attachments transmitted together with it, = is >> > intended for the use only of the person (or persons) to whom it is >> > addressed. It may contain information which is confidential and/or >> > protected by legal privilege. Accordingly, any dissemination, >> distribution, >> > copying or use of this message, or any part of it or anything sent >> together >> > with it, other than by intended recipients, may constitute a breach of >> > civil or criminal law and is hereby prohibited. Unless otherwise state= d, >> > any views expressed in this message are those of the person sending it >> and >> > not the sender's employer. No responsibility, legal or otherwise, of >> > whatever nature, is accepted as to the accuracy of the contents of thi= s >> > message or for the completeness of the message as received. Anyone who >> is >> > not the intended recipient of this message is advised to make no use o= f >> it >> > and is requested to contact Featurespace Limited as soon as possible. >> Any >> > recipient of this message who has knowledge or suspects that it may ha= ve >> > been the subject of unauthorised interception or alteration is also >> > requested to contact Featurespace Limited. >> > >> > --00000000000053655705a857eda0--