Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
Sender: ismaelj@gmail.com
In-Reply-To: <29541B46-E1EE-46A7-9FD1-195DF2465519@apache.org>
References: <6D71A36A-B36D-4439-B1F9-9858AC0227D3@jordanzimmerman.com>
 <586891CD-7939-47A4-B8E5-7850A1BBE8D7@apache.org> <D3884C0B.46D25%cnauroth@hortonworks.com>
 <C4FA975B-BF1C-4B77-8A67-5A2ED03AF80F@jordanzimmerman.com>
 <CAD5tkZb9uTmRyE4bv-a=uY1O-93CBCAk85ps_yCBEeyaFB-7+Q@mail.gmail.com> <29541B46-E1EE-46A7-9FD1-195DF2465519@apache.org>
From: Ismael Juma <ismael@juma.me.uk>
Date: Sat, 18 Jun 2016 21:21:04 +0200
Message-ID: <CAD5tkZa+nKe1GeGdEtmnS9S-42u2mHVeWsVZwhOYL00gg8rB1Q@mail.gmail.com>
Subject: Re: FYI - Apache ZooKeeper Backup, a Treatise
To: dev@kafka.apache.org
Cc: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=001a114b2f80b551df05359261f8
archived-at: Sat, 18 Jun 2016 19:21:54 -0000

--001a114b2f80b551df05359261f8
Content-Type: text/plain; charset=UTF-8

Hi Flavio,

Comments below.

On Sat, Jun 18, 2016 at 4:50 PM, Flavio Junqueira <fpj@apache.org> wrote:

> For acls, you can simply re-run the acl command to re-introduce them,
> unless you assume that no record of acls is maintained once they are
> introduced. If that's the case, then another way is to simply read
> periodically the zk state and keep that information somewhere else to be
> extra safe. This seems easier than dealing with raw zk backups.


Yes, I think it's not uncommon for users not to have a separate record of
Kafka ACLs. It sounds like we need to improve our documentation to cover
this area.

For topic configs, we would need to contact servers to reconstruct the data.
>

That makes sense although the mechanism to do that is currently missing as
far as I know.

It is important to keep in mind that this happens quite rarely, though, and
> if such a daunting scenario does happen, it is quite possible that
> recovering the zk state is the least important of our problems. If you do
> worry about losing too many replicas of anything, be it zk or kafka
> brokers, to the point of not being able to recover, then it is indeed
> important to have a plan to restore data. Typically we try to avoid these
> scenarios by having enough replicas and making sure that we reduce the
> chance of correlated events (e.g., by having remote replicas, rack
> awareness), for some definition of enough.
>

Yes, agreed.

As discussed offline, another scenario where backups (whether raw or by
reading ZK state) or delayed replicas can help is in case of user error. An
admin may incorrectly delete all ACLs, for example. One would hope this
never happens, but apparently it does (a UK ISP once deleted gigabytes of
customers' emails due to an admin error).

Ismael

--001a114b2f80b551df05359261f8--