zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norbert Kalmar <nkal...@cloudera.com.INVALID>
Subject Re: Code to clean up transaction logs needs snapshots before it works?
Date Wed, 14 Aug 2019 12:55:24 GMT
Read doesn't matter, there will be no txn logs for read. Only modifications
are logged (state changes to the datatree).

As for why all txn logs are 64MB:
"preAllocSize
(Java system property: zookeeper.preAllocSize)
To avoid seeks ZooKeeper allocates space in the transaction log file in
blocks of preAllocSize kilobytes. The default block size is 64M. One reason
for changing the size of the blocks is to reduce the block size if
snapshots are taken more often. (Also, see snapCount)."

Note that all txn is written to one log file. That's why it is 64MB by
default.
I didn't check which version of admin guide I quoted, but here is the one
from 3.4.13:

snapCount
ZooKeeper records its transactions using snapshots and a transaction log
(think write-ahead log).The number of transactions recorded in the
transaction log before a snapshot can be taken (and the transaction log
rolled) is determined by snapCount. In order to prevent all of the machines
in the quorum from taking a snapshot at the same time, each ZooKeeper
server will take a snapshot when the number of transactions in the
transaction log reaches a runtime generated random value in the
[snapCount/2+1, snapCount] range.The default snapCount is 100,000.

I'm not sure without checking the code what happens if preAllocSize is
filled (new log file I think), but after a restart, there will be a new log
file created. No way to continue the last one. That's why there could be
multiple log files, and no transaction. But if you haven't reached the
100.000 transactions (writes that is, reads doesn't count), there will be
no snapshot.

Regards,
Norbert

On Wed, Aug 14, 2019 at 2:35 PM Koen De Groote <koen.degroote@limecraft.com>
wrote:

> >Without a snapshot, we cannot delete the log files, as we would have no
> >means of recovery. txn logs applied to the snapshot gives us back the
> >state. Without snapshot, all txn logs needs to be "replayed" in a
> recovery.
> >And you need all the log files created since your last snapshot (in this
> >case, all the txn logs as there were no snapshots yet).
>
> Makes sense.
>
> In the heaviest environment I'm hitting around 200 requests per second,
> which all have to get data from zookeeper. Not sure what the impact is in
> terms of snapCount, I didn't set up the system myself and don't fully grasp
> the internals.
>
> As for the snapCount, that hasn't been touched, so that will be the default
> in my environments.
> I've read that the log files are preallocated. I see them as being 65MB a
> piece.
>
> Which makes me wonder: how many of those until the process hits 100000
> snaps?
>
> auto-cleaning is setup, retainCount=3, purgeInterval=1
>
> Restarting the zookeeper process shouldn't affect this count, I think? It
> doesn't happen often, though it might on test environments.
>
>
>
> On Wed, Aug 14, 2019 at 2:06 PM Norbert Kalmar
> <nkalmar@cloudera.com.invalid>
> wrote:
>
> > Hi,
> >
> > Without a snapshot, we cannot delete the log files, as we would have no
> > means of recovery. txn logs applied to the snapshot gives us back the
> > state. Without snapshot, all txn logs needs to be "replayed" in a
> recovery.
> > And you need all the log files created since your last snapshot (in this
> > case, all the txn logs as there were no snapshots yet).
> >
> > As for why there is no snapshot. What is your load? Per the admin guide:
> >
> > "snapCount
> > (Java system property: zookeeper.snapCount)
> > ZooKeeper logs transactions to a transaction log. After snapCount
> > transactions are written to a log file a snapshot is started and a new
> > transaction log file is created. The default snapCount is 100,000."
> >
> > By default there will be no auto-cleaning of the snapshot and log files.
> > Check the autopurge.snapRetainCount and autopurge.purgeInterval settings
> > for this.
> >
> > Regards,
> > Norbert
> >
> > On Wed, Aug 14, 2019 at 1:21 PM Koen De Groote <
> > koen.degroote@limecraft.com>
> > wrote:
> >
> > > Greetings all.
> > >
> > > I was debugging something an ran into this bit of code:
> > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/PurgeTxnLog.java#L81
> > >
> > > If I understand it correctly, it seems that this means log.xxxxx files
> > will
> > > only get deleted if there's a snapshot.
> > >
> > > Which is troublesome, as my dataDir is filling up with log files but
> not
> > a
> > > single snapshot in sight.
> > >
> > > 1: Is this correct behavior? Both the logic of needing a snapshot and
> the
> > > fact that not snapshots are being generated?
> > >
> > > 2: While not having a fix for this, what would be useful to know is:
> can
> > > these log files be freely deleted? Or does the most recent one need to
> be
> > > kept, or how does it go with this files?
> > >
> > > I thought that running "bin/zkCleanup.sh /data -n 3" would clean up
> both
> > > the snapshots and the logs, but it appears that if there are not
> > snapshots,
> > > the logs aren't cleaned either.
> > >
> > > What are my options here?
> > >
> > > Kind regards,
> > > Koen De Groote
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message