zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Cooper <tom.n.coo...@gmail.com>
Subject Snapshot creation after 3.4.14 -> 3.5.6 upgrade
Date Mon, 09 Dec 2019 10:10:54 GMT
Hi,

I am working on upgrading a Kubernetes based 3.4.14 cluster to 3.5.6.

I are hitting the issue where the new snapshot checks prevent the newly
upgraded 3.5.6 cluster from starting. This usually happens in our test
set-up as the 3.4.14 cluster is not very old and has not hit the snapCount
limit yet. Typically only one node (presumable the leader) has a snapshot
file but the other do not.

I can disable the snapshot checks via the config value for the upgrade to
3.5.6, however we obviously need to re-enable the checks ASAP. The issue is
making sure all of the ZK nodes have snapshots so that the checks can be
re-enabled.

My question is: Is there a way to force the snapshot creation / sync from
the leader?

As part of our upgrade process we need to restart the servers soon after
the initial upgrade of the ZK container images to enable config changes. If
the snapshot creation / sync hasn't happened by this point then obviously
the checks will fail. If we can force a snapshot creation, after the
initial upgrade, we can re-enable the checks as part of the upgrade
process. If not, we will need to create some method to check if the
snapshots have been created and periodically inspect the ZK server file
system, re-enabling the checks once they all have snapshots.

Any pointers would be appreciated.

Thanks in advance,

Tom Cooper

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message