Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of fpjunqueira@yahoo.com
 designates 98.139.213.161 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\))
Subject: Re: Transaction logs and snapshots
From: Flavio Junqueira <fpjunqueira@yahoo.com.INVALID>
In-Reply-To: <1458C6B4EF2B37428F20085D75DD94DB0179DE4369@MARY.main.mobik.si>
Date: Thu, 16 Apr 2015 21:29:23 +0100
Cc: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <6F73A35C-6BF8-4B32-9BD6-0B46D3449BD3@yahoo.com>
References: <1458C6B4EF2B37428F20085D75DD94DB0179DE3F9B@MARY.main.mobik.si>
 <1052245806.5125456.1429177740571.JavaMail.yahoo@mail.yahoo.com>
 <1458C6B4EF2B37428F20085D75DD94DB0179DE4369@MARY.main.mobik.si>
To: Dejan Markic <Dejan.Markic@mobik-ics.com>

Another think you could do is to make snapCount very large so that =
snapshots are created infrequently. But, let me step back and ask you =
why you think ZK is a good fit for your project. It isn't clear to me =
that your case is a good one for ZK.

-Flavio

=20
> On 16 Apr 2015, at 11:01, Dejan Markic <Dejan.Markic@mobik-ics.com> =
wrote:
>=20
> Hello!
>=20
> Log seems to be always 67.108.880 bytes.
> Snapshots are currently between 30-40MB. Snapshot is created almost =
every minute.
> Yes, data durability is not important at all. Once the session ends =
(it may last between 0 and few minutes, average around 1-2 minutes =
maybe), I don't need it anymore. I regulary remove  nodes that are not =
changed for more than 10 minutes.
> I even recieve updates for sessions, so even if ZK looses data, I =
would get it back after few minutes.
>=20
> Thanks!
>=20
> Kind regards,
> Dejan
>=20
>=20
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]=20
> Sent: Thursday, April 16, 2015 11:49 AM
> To: user@zookeeper.apache.org
> Subject: Re: Transaction logs and snapshots
>=20
> Hi Dejan,
> For a typical ZK application, granularity of hours is more than =
enough, since it is supposed to be an infrequent background task. In =
your case, it sounds like durability isn't an important property because =
if it is you shouldn't be getting rid of disk data this fast. I'm also =
wondering about the amount of data you're generating. What's the size of =
your snapshots and txn logs?
> -Flavio=20
>=20
>=20
>     On Thursday, April 16, 2015 10:26 AM, Dejan Markic =
<Dejan.Markic@mobik-ics.com> wrote:
>=20
>=20
>=20
> Hello Flavio!
>=20
> Would that mean, that zkCleanup.sh would not be needed?=20
> PurgeInterval is minimum 1 hour? Why is it so high?
>=20
> Thanks!
>=20
> Kind regards,
> Dejan Markic
>=20
>=20
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]=20
> Sent: Thursday, April 16, 2015 11:15 AM
> To: user@zookeeper.apache.org
> Subject: Re: Transaction logs and snapshots
>=20
> Hi Dejan,
> Check if the autopurge feature solves your problem:
> =
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_advancedConf=
iguration
>=20
> -Flavio=20
>=20
>=20
>     On Thursday, April 16, 2015 9:17 AM, Dejan Markic =
<Dejan.Markic@mobik-ics.com> wrote:
>  =20
>=20
>=20
> Hello all!
>=20
> We are running 3 ZK servers in ensemble, and ZK is processing a lot of =
commands per seconds. There are probably around 300 nodes =
created/checked/set/get per second.
> Since we have only information about live sessions we handle in ZK, we =
don't need any data persistency - eg: we can stop all nodes, clean all =
transaction logs/snapshots, and start them up again, without any issues.
> Since we have a lot of requests/changes, we have moved dataDir onto =
ramdisk, so we have no problems with disk IOPS, etc.
> Is there a way, to minimze the usage of snapshots/logs so ramdisk =
would not get filled up? It happens that transaction logs/snapshots grow =
so large, that we run out of space on ramdisk.
> We issue >/usr/share/zookeeper/bin/zkCleanup.sh -n 3< every 2 minutes, =
so this should cleanup the dataDir quite often. Why is >count number of =
snapshots/logs to keep< limited to 3 and not below?
> I assume, in my setup, I don't even need snapshots/logs to be stored =
after they are not actively needed?=20
> So my basic questions are:
> - can I somehow get rid of snapshot/logs sooner, more often ... ?
> - when is snapshot created? Can it be created sooner, so it would be =
smaller?=20
> - Is it possible to get rid of snapshot/logs all together?
>=20
> Thank you for all your inputs and kind regards, Dejan Markic
>=20
>=20
>  =20
>=20
>=20
>=20
>=20