From user-return-12444-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Sun Dec 15 22:53:48 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 802E3180636 for ; Sun, 15 Dec 2019 23:53:48 +0100 (CET) Received: (qmail 85439 invoked by uid 500); 15 Dec 2019 22:53:47 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 85426 invoked by uid 99); 15 Dec 2019 22:53:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Dec 2019 22:53:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 36029C20A4 for ; Sun, 15 Dec 2019 22:53:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id bbsdMumKv37O for ; Sun, 15 Dec 2019 22:53:42 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.208.181; helo=mail-lj1-f181.google.com; envelope-from=shralex@gmail.com; receiver= Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 45AE2BC509 for ; Sun, 15 Dec 2019 22:53:42 +0000 (UTC) Received: by mail-lj1-f181.google.com with SMTP id k8so4670553ljh.5 for ; Sun, 15 Dec 2019 14:53:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ItxSvV5cnMO1gTKVUKOeBtxB0kCcEjjg2wENL9o9zoA=; b=BWXf5tpJa1mC4fNNo1eaBLp7Tq7q7ZX4qUWLpb0AI0oou5Stz3oE5AHev9WMV8Qg9/ U3ImkF4mGPXFcpS1eSlMGGsAzwujfccpG3XjnIXp3tQLGTv/O0sNy6zLeJejnTvepDdj Qo5/KN3XbNOYGpYxVv03sKUovQd/B+Jd1Yv5cMaT1a7A2pBbM2XY5YL35KW7pzAAV9PO PuTVQtlAWgxp8LXMV23r8BgWvCn3N86YLvMYXhD9tPEp51rVbxm6bvkpq5KENUztkBHA c3cDjOZ+9q5bCQCF0qNIiBn9MAnSi0uBBkop+MZQ691C6G0S1FkYlXhc03i8dZsgkdcB jU6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ItxSvV5cnMO1gTKVUKOeBtxB0kCcEjjg2wENL9o9zoA=; b=KvoPr1AMqXjS9GQqwBzJRaMNI7rurCAr61Xi+eRxp2mcrbtES4xzeoD5s0a3d/c3AD H81qRNpR6yxJADglDHjC0E2IEaI/xMrY3SoMPx9OmPX4fVzIBgKTUE+Ipz0w2dfiT3yE 5W+wwFqkK28X6ChGwVBTPNJUln47sv61JgLOe46Nn9XW1JFsoPacD9y3hVkE910924Az R9BBblWwHIwEEqX7xOWDGn93/jJ4EFmhe0rdVa6ny0LwzAA9/qbeo0/v0PMHRD2OqT8J 1Cqeu+eHIVydQRqkM6VrwC0rcYBaBS/SjhtdNQgABNoUNzHoJeIm/OHo66eY2CLs16iS DXVg== X-Gm-Message-State: APjAAAUXPlrTmFa/DzZhuFdw7ePN8AjbFdWHrjI9c0UXBdzGG1M4obOV ihAocWSw/nSNHkoe/A81l2mnTo5kmw6kul/sf3ooP7yQ X-Google-Smtp-Source: APXvYqwojMXJbm0GmQhvtv0Es1qaMf21CKTSmFu11lYrOXTuYE+HAP7yqf5VetiTUondXeXWcdE6KCs5WpaMRUmGEmI= X-Received: by 2002:a2e:95c4:: with SMTP id y4mr17587621ljh.38.1576450414466; Sun, 15 Dec 2019 14:53:34 -0800 (PST) MIME-Version: 1.0 References: <7a4d6b9e-27c1-6e57-462e-286d04984766@ish.com.au> <048f3e17be5145e39e09ca421905c307@dlr.de> <7996f33a-250b-d679-8bc9-9091450b4e6e@ish.com.au> <2a235e15-1d0a-7a12-0da7-660bf0475193@ish.com.au> In-Reply-To: <2a235e15-1d0a-7a12-0da7-660bf0475193@ish.com.au> From: Alexander Shraer Date: Sun, 15 Dec 2019 14:53:20 -0800 Message-ID: Subject: Re: AW: Configuration management for zoo.cfg To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="000000000000f5dc130599c5f61b" --000000000000f5dc130599c5f61b Content-Type: text/plain; charset="UTF-8" I wasn't sure whether extracting such information from the log is simple, and since reconfigurations may impact the cluster in significant ways (or in the extreme bring it down completely) an easily accessible record seemed good to have, at least for debugging. I agree that this can be made configurable, and would also not mind very much not having a history at all, if others don't find it very useful. However this is a breaking change so probably requires more people to chime in. > In case of some network issue, where a node repeatedly flaps, why would you want to fill the directory with possibly thousands of files? Automating reconfigations was not part of the release, only the basic mechanism was provided and not for example the policy of when you'd want to reconfigure and what changes to do. But I agree that an automatic system like that should take care of this situation. Alex On Sun, Dec 15, 2019 at 2:26 PM Aristedes Maniatis wrote: > Can you explain a bit more about the use-case for when you'd want to > keep the history of the dynamic file. Surely the log file will contain > information about peers joining and leaving the cluster and is easier to > parse if you care about tracking that sort of thing. > > In case of some network issue, where a node repeatedly flaps, why would > you want to fill the directory with possibly thousands of files? > > > Ari > > > On 15/12/19 2:35pm, Alexander Shraer wrote: > > Hi Ari, > > > > Yes, you're totally right about the design goals. > > > > A mode where historic files aren't preserved could be useful. This > > could perhaps be added to the static config file as a parameter. > > > > Alternatively / in addition, maybe we could slightly change the way > history > > is staved. I don't really like the fact that we're actually using > > the file name to determine the version of the config (rather than > > information inside the file), this is used internally in ZK to decide > which > > config to use (the one with higher number wins). > > This method could fix this issue as well: > > - File name always stays the same, addressing your problem, and we don't > > need to edit the static config file every time. > > - Dynamic config file contains the config version as a key. > > - Before overwriting the dynamic config file, we store a file with the > > previous config, including the version in the file name. > > > > This would change the current behavior a bit, hopefully no one is relying > > on the file name to contain the version. > > > > This should not be difficult to implement, would you like to open a Jira > > and take a stab at implementing it ? I can review it. > > > > Something to notice about the "version" of the config - currently when > the > > config is stored in memory, it appears as a key in the configuration. > When > > stored in the temporary config file (pre-commit), it appears as an > explicit > > key, but when committed it does not appear inside the dynamic file - only > > in the file name.This is controlled by the last argument of > > QuorumPeerConfig.writeDynamicConfig. > > > > See also QuorumPeerConfig.java parse() parseProperties() etc and > > QuorumPeer.java setQuorumVerifier(). > > > > Thanks, > > Alex > > > > On Sat, Dec 14, 2019 at 6:32 PM Aristedes Maniatis > wrote: > > > >> Will anything bad happen if I make the config file read-only for > >> zookeeper? I assume the design goals here were: > >> > >> * atomic rewrites of the dynamic config, preserving historic files > >> > >> * ability for zookeeper to know which was the most recent config file on > >> restart > >> > >> > >> Those goals are a bit unnecessary for me. I don't really care about > >> historic configuration, so just writing to a temp file and moving over > >> the existing one would work great. Alternatively tracking the current > >> file in memory without rewriting the zoo.cfg would also be great, since > >> I don't care about the effort on startup to rediscover peers. > >> > >> Is there a way to get Zookeeper to play better with not rewriting its > >> own config file for my use case? > >> > >> > >> Ari > >> > >> > >> On 12/12/19 5:53am, Alexander Shraer wrote: > >>> It will change, the number represents the version of the configuration, > >> and > >>> will be updated if you issue a reconfiguration command. Its basically > the > >>> zxid of the command. > >>> > >>> > >>> Alex > >>> > >>> On Tue, Dec 10, 2019 at 11:25 PM Aristedes Maniatis > >> wrote: > >>>> On 11/12/19 6:21pm, Arne.Bachmann@dlr.de wrote: > >>>>> Hey Ari, > >>>>> > >>>>> I directly used the filename zoo.cfg.dynamic.100000000 and never > >>>> had a > >>>>> problem. > >>>>> Arne > >>>> Hmmm... that's an oddly obvious answer. I just assumed the 100000000 > >>>> would change randomly. What's even the point of it? > >>>> > >>>> > >>>> Ari > >>>> > >>>> > --000000000000f5dc130599c5f61b--