zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koen De Groote <koen.degro...@limecraft.com>
Subject Re: Code to clean up transaction logs needs snapshots before it works?
Date Wed, 14 Aug 2019 15:02:47 GMT
Al right, I'll try and go forward with that info. Thanks.

On Wed, Aug 14, 2019 at 2:55 PM Norbert Kalmar <nkalmar@cloudera.com.invalid>
wrote:

> Read doesn't matter, there will be no txn logs for read. Only modifications
> are logged (state changes to the datatree).
>
> As for why all txn logs are 64MB:
> "preAllocSize
> (Java system property: zookeeper.preAllocSize)
> To avoid seeks ZooKeeper allocates space in the transaction log file in
> blocks of preAllocSize kilobytes. The default block size is 64M. One reason
> for changing the size of the blocks is to reduce the block size if
> snapshots are taken more often. (Also, see snapCount)."
>
> Note that all txn is written to one log file. That's why it is 64MB by
> default.
> I didn't check which version of admin guide I quoted, but here is the one
> from 3.4.13:
>
> snapCount
> ZooKeeper records its transactions using snapshots and a transaction log
> (think write-ahead log).The number of transactions recorded in the
> transaction log before a snapshot can be taken (and the transaction log
> rolled) is determined by snapCount. In order to prevent all of the machines
> in the quorum from taking a snapshot at the same time, each ZooKeeper
> server will take a snapshot when the number of transactions in the
> transaction log reaches a runtime generated random value in the
> [snapCount/2+1, snapCount] range.The default snapCount is 100,000.
>
> I'm not sure without checking the code what happens if preAllocSize is
> filled (new log file I think), but after a restart, there will be a new log
> file created. No way to continue the last one. That's why there could be
> multiple log files, and no transaction. But if you haven't reached the
> 100.000 transactions (writes that is, reads doesn't count), there will be
> no snapshot.
>
> Regards,
> Norbert
>
> On Wed, Aug 14, 2019 at 2:35 PM Koen De Groote <
> koen.degroote@limecraft.com>
> wrote:
>
> > >Without a snapshot, we cannot delete the log files, as we would have no
> > >means of recovery. txn logs applied to the snapshot gives us back the
> > >state. Without snapshot, all txn logs needs to be "replayed" in a
> > recovery.
> > >And you need all the log files created since your last snapshot (in this
> > >case, all the txn logs as there were no snapshots yet).
> >
> > Makes sense.
> >
> > In the heaviest environment I'm hitting around 200 requests per second,
> > which all have to get data from zookeeper. Not sure what the impact is in
> > terms of snapCount, I didn't set up the system myself and don't fully
> grasp
> > the internals.
> >
> > As for the snapCount, that hasn't been touched, so that will be the
> default
> > in my environments.
> > I've read that the log files are preallocated. I see them as being 65MB a
> > piece.
> >
> > Which makes me wonder: how many of those until the process hits 100000
> > snaps?
> >
> > auto-cleaning is setup, retainCount=3, purgeInterval=1
> >
> > Restarting the zookeeper process shouldn't affect this count, I think? It
> > doesn't happen often, though it might on test environments.
> >
> >
> >
> > On Wed, Aug 14, 2019 at 2:06 PM Norbert Kalmar
> > <nkalmar@cloudera.com.invalid>
> > wrote:
> >
> > > Hi,
> > >
> > > Without a snapshot, we cannot delete the log files, as we would have no
> > > means of recovery. txn logs applied to the snapshot gives us back the
> > > state. Without snapshot, all txn logs needs to be "replayed" in a
> > recovery.
> > > And you need all the log files created since your last snapshot (in
> this
> > > case, all the txn logs as there were no snapshots yet).
> > >
> > > As for why there is no snapshot. What is your load? Per the admin
> guide:
> > >
> > > "snapCount
> > > (Java system property: zookeeper.snapCount)
> > > ZooKeeper logs transactions to a transaction log. After snapCount
> > > transactions are written to a log file a snapshot is started and a new
> > > transaction log file is created. The default snapCount is 100,000."
> > >
> > > By default there will be no auto-cleaning of the snapshot and log
> files.
> > > Check the autopurge.snapRetainCount and autopurge.purgeInterval
> settings
> > > for this.
> > >
> > > Regards,
> > > Norbert
> > >
> > > On Wed, Aug 14, 2019 at 1:21 PM Koen De Groote <
> > > koen.degroote@limecraft.com>
> > > wrote:
> > >
> > > > Greetings all.
> > > >
> > > > I was debugging something an ran into this bit of code:
> > > >
> > > >
> > >
> >
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/PurgeTxnLog.java#L81
> > > >
> > > > If I understand it correctly, it seems that this means log.xxxxx
> files
> > > will
> > > > only get deleted if there's a snapshot.
> > > >
> > > > Which is troublesome, as my dataDir is filling up with log files but
> > not
> > > a
> > > > single snapshot in sight.
> > > >
> > > > 1: Is this correct behavior? Both the logic of needing a snapshot and
> > the
> > > > fact that not snapshots are being generated?
> > > >
> > > > 2: While not having a fix for this, what would be useful to know is:
> > can
> > > > these log files be freely deleted? Or does the most recent one need
> to
> > be
> > > > kept, or how does it go with this files?
> > > >
> > > > I thought that running "bin/zkCleanup.sh /data -n 3" would clean up
> > both
> > > > the snapshots and the logs, but it appears that if there are not
> > > snapshots,
> > > > the logs aren't cleaned either.
> > > >
> > > > What are my options here?
> > > >
> > > > Kind regards,
> > > > Koen De Groote
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message