zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norbert Kalmar <nkal...@cloudera.com.INVALID>
Subject Re: Code to clean up transaction logs needs snapshots before it works?
Date Wed, 14 Aug 2019 12:06:05 GMT
Hi,

Without a snapshot, we cannot delete the log files, as we would have no
means of recovery. txn logs applied to the snapshot gives us back the
state. Without snapshot, all txn logs needs to be "replayed" in a recovery.
And you need all the log files created since your last snapshot (in this
case, all the txn logs as there were no snapshots yet).

As for why there is no snapshot. What is your load? Per the admin guide:

"snapCount
(Java system property: zookeeper.snapCount)
ZooKeeper logs transactions to a transaction log. After snapCount
transactions are written to a log file a snapshot is started and a new
transaction log file is created. The default snapCount is 100,000."

By default there will be no auto-cleaning of the snapshot and log files.
Check the autopurge.snapRetainCount and autopurge.purgeInterval settings
for this.

Regards,
Norbert

On Wed, Aug 14, 2019 at 1:21 PM Koen De Groote <koen.degroote@limecraft.com>
wrote:

> Greetings all.
>
> I was debugging something an ran into this bit of code:
>
> https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/server/PurgeTxnLog.java#L81
>
> If I understand it correctly, it seems that this means log.xxxxx files will
> only get deleted if there's a snapshot.
>
> Which is troublesome, as my dataDir is filling up with log files but not a
> single snapshot in sight.
>
> 1: Is this correct behavior? Both the logic of needing a snapshot and the
> fact that not snapshots are being generated?
>
> 2: While not having a fix for this, what would be useful to know is: can
> these log files be freely deleted? Or does the most recent one need to be
> kept, or how does it go with this files?
>
> I thought that running "bin/zkCleanup.sh /data -n 3" would clean up both
> the snapshots and the logs, but it appears that if there are not snapshots,
> the logs aren't cleaned either.
>
> What are my options here?
>
> Kind regards,
> Koen De Groote
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message