zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Baskar Duraikannu <baskar.duraika...@outlook.com>
Subject Re: Problem with ZK log files and snapshots
Date Tue, 17 Jul 2018 11:12:26 GMT
If your transactions are short lived (something like you are taking a lock and then releasing
it), you will find that “containers” holding these locks are not cleaned by Zookeeper.
This in turn will make snapshot files big and will cause all sorts of issues. I found myself
in that situation few months back. New version of zookeeper has container support which solves
this issue.  We had this issue and upgrading to 3.5.4-Beta and appropriate version of curator
solved the issue. 

Good luck.

> On Jul 16, 2018, at 7:04 AM, Jostein Elvaker Haande <jehaande@gmail.com> wrote:
> 
> Hello,
> 
> I'm using ZooKeeper 3.4.12, and I'm facing an issue where the ZK log files
> keep up eating disk space. I'm running this on several machines, and the
> symptoms I'm seeing are the seem:
> 
> - I see a lot of log files
> - Rarely, if ever any snapshot files
> 
> I started digging into it, and from
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fzookeeper.apache.org%2Fdoc%2Fr3.4.12%2FzookeeperAdmin.html%23sc_maintenance&data=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636673358550132623&sdata=pBKODIXN1Yl5%2BL7zAtWvgoocWK20g8BAbFygPFpa648%3D&reserved=0
> one should use the ZK Java class org.apache.zookeeper.server.PurgeTxnLog
> for this. However, seeing as I rarely see any snapshot files, PurgeTxnLog
> never performs any cleanup.
> 
> Going deeper down, I can see from the documentation that a snapshot is
> created every 100K transactions that are kept in the log files, so I was
> interested in seeing how man transaction my logs kept. I then made a script
> that used the Java class org.apache.zookeeper.server.LogFormatter which
> outputs number of transactions on the same line, and I noticed that my ZK
> instances saw a very low transaction count. For most of my instances, the
> transaction count was below 20K.
> 
> Now I know that by setting snapCount to a value lower than the default
> 100K, it should generate snapshots more frequently, and ZK should then be
> able to do this cleanup automatically which is controlled
> by autopurge.snapRetainCount and autopurge.purgeInterval.
> 
> However, when setting a new value for snapCount, this does not seem to have
> any effect on the existing transaction log files. So my question is, how do
> I deal with the old transaction logs? How do I proceed to clean this away
> safely?
> 
> -- 
> Yours sincerely Jostein Elvaker Haande
> "A free society is a society where it is safe to be unpopular"
> - Adlai Stevenson
> 
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftolecnal.net&data=02%7C01%7C%7Ca930ebc3267246f355f208d5eb0bddf9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636673358550132623&sdata=kUcc%2BHDxvoBG%2F7D3HQx2XxsPHfDoQmkRtXH6fVlN5To%3D&reserved=0
-- tolecnal at tolecnal dot net

Mime
View raw message