From user-return-12396-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Mon Dec 9 10:11:35 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 24B5218064C for ; Mon, 9 Dec 2019 11:11:35 +0100 (CET) Received: (qmail 78074 invoked by uid 500); 9 Dec 2019 10:11:33 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 78059 invoked by uid 99); 9 Dec 2019 10:11:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Dec 2019 10:11:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id C0242C20AC for ; Mon, 9 Dec 2019 10:11:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id NriemjDksz-l for ; Mon, 9 Dec 2019 10:11:31 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.194; helo=mail-qk1-f194.google.com; envelope-from=tom.n.cooper@gmail.com; receiver= Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 81DE7BC509 for ; Mon, 9 Dec 2019 10:11:31 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id z14so2191404qkg.9 for ; Mon, 09 Dec 2019 02:11:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FkpRoZPQtjno3UrH6USExGgJD2r5PT7HNZL+JvsqRD4=; b=ajS3HGo11t7tziT8VJguZf+S8XLobmi/jl4CV4haWqDWlwdiJTlolEOp/zPnPKrm2Q vk3w+RRNzDHzskpKTVChmC6eS5Az7eBEHH80QY1uR2FZcSVEAXLOu/SQfLf/6nzwRJmG 5sP341C8rQZbSE3uo9mPtJufmSCV/dBW9XVPU5Ihy72DexDPgHuz5LAw1rdWC8JO9BSm eUmMZtPGW8fP40l3HO+En5dA4n1eB82SZfQm94NmzpYc2az6UzPyHgjsa0VzByPZhq7k 4qS+VlCM9kWaZPrMv1V3f/7cQvEt6wxidO8FoYaOXiVoUvfVKecx0HyWdKYOc6yrpIb6 nrKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FkpRoZPQtjno3UrH6USExGgJD2r5PT7HNZL+JvsqRD4=; b=syfL8fNQJuZyl37sZAqbxQCEFy/ZE3EYa9muqL/Jx7TTKn/Q5BiznSPi6kXaXiNecy ZOS5IJ6/ww1lKN1BOne/HYzXI+fSWqTwlAKjK4xdlRKfq5SXeU8z5ISq9tuLsm2CKPoN KqkdTsNkAwdA/Azi1SIpr/dG5WQsplgopAupQGE2hMcpf3WIN4BOcU8RBbjEwZ+OMi5R yHk0OPXf4HxY4jRa/RmwBsUqDaFO7gdQRiftu44UhSQKTjZ6i1JTrxuJzBEg/mZ0279C 9A0b0udcuDWvg99HbOtigNHLAFs1bEv6/6cBO96nodn1QtZsve/k3W4vG1rHTx1+86D/ YpbA== X-Gm-Message-State: APjAAAUITesnson3O822O6N+hPaoFtZbARSIYKPYdgfeCS0FHvOR9Wlw mBCn5EbR7fU7B5i7lZ1umPEdSBhWziL/gJ2gXRule0TrcJI= X-Google-Smtp-Source: APXvYqwI42xrs7nPRI6MrfXXu6kfuZKw7n7d7JVMpqZMx5tL9C7C4mPY2JJH/eQ8wZq27t/SkzLiVBg9j3c75Sl7j0Y= X-Received: by 2002:a37:4d16:: with SMTP id a22mr26713798qkb.237.1575886290477; Mon, 09 Dec 2019 02:11:30 -0800 (PST) MIME-Version: 1.0 From: Thomas Cooper Date: Mon, 9 Dec 2019 10:10:54 +0000 Message-ID: Subject: Snapshot creation after 3.4.14 -> 3.5.6 upgrade To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="0000000000008ce5cd0599429ec3" --0000000000008ce5cd0599429ec3 Content-Type: text/plain; charset="UTF-8" Hi, I am working on upgrading a Kubernetes based 3.4.14 cluster to 3.5.6. I are hitting the issue where the new snapshot checks prevent the newly upgraded 3.5.6 cluster from starting. This usually happens in our test set-up as the 3.4.14 cluster is not very old and has not hit the snapCount limit yet. Typically only one node (presumable the leader) has a snapshot file but the other do not. I can disable the snapshot checks via the config value for the upgrade to 3.5.6, however we obviously need to re-enable the checks ASAP. The issue is making sure all of the ZK nodes have snapshots so that the checks can be re-enabled. My question is: Is there a way to force the snapshot creation / sync from the leader? As part of our upgrade process we need to restart the servers soon after the initial upgrade of the ZK container images to enable config changes. If the snapshot creation / sync hasn't happened by this point then obviously the checks will fail. If we can force a snapshot creation, after the initial upgrade, we can re-enable the checks as part of the upgrade process. If not, we will need to create some method to check if the snapshots have been created and periodically inspect the ZK server file system, re-enabling the checks once they all have snapshots. Any pointers would be appreciated. Thanks in advance, Tom Cooper --0000000000008ce5cd0599429ec3--