Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of fpjunqueira@yahoo.com
 designates 98.139.213.144 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\))
Subject: Re: Transaction logs and snapshots
From: Flavio Junqueira <fpjunqueira@yahoo.com.INVALID>
In-Reply-To: <1458C6B4EF2B37428F20085D75DD94DB0179DF4039@MARY.main.mobik.si>
Date: Thu, 16 Apr 2015 22:26:25 +0100
Cc: "user@zookeeper.apache.org" <user@zookeeper.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <9B2884F2-78E2-4178-BC3A-2EC705A4D61D@yahoo.com>
References: <1458C6B4EF2B37428F20085D75DD94DB0179DE3F9B@MARY.main.mobik.si>
 <1052245806.5125456.1429177740571.JavaMail.yahoo@mail.yahoo.com>
 <1458C6B4EF2B37428F20085D75DD94DB0179DE4369@MARY.main.mobik.si>
 <,<6F73A35C-6BF8-4B32-9BD6-0B46D3449BD3@yahoo.com> <>>
 <1458C6B4EF2B37428F20085D75DD94DB0179DF4039@MARY.main.mobik.si>
To: Dejan Markic <Dejan.Markic@mobik-ics.com>

Distributed locks is indeed part of our bread and butter. Why don't you =
want to write to disk? Your workload does't seem to be heavy. Does the =
IO traffic compete with some other traffic you have?

-Flavio

> On 16 Apr 2015, at 22:15, Dejan Markic <Dejan.Markic@mobik-ics.com> =
wrote:
>=20
> Hello Flavio!
>=20
> Yes, indeed, ZK might not be the best option - but I could not find =
any better. What we need is a rather fast, distributed locking "system". =
ZK was at the moment the best option, and after testing it seemed to be =
the thing we are looking for. Other than snapshots/transaction logs, we =
have no problems. It easily handles our current load. It has C library, =
which makes it fairly easy to port it to other software.
> What we need (but I cannot find any) is distributed in-memory =
distributed locking system where we can store some small information.=20
> For instance, we use ZK's nodes as /SESSION_ID ... we lock it here, =
and then we use eg /SESSION_ID/my_var to store something. After session =
is gone, we remove this node and all information about it.
>=20
> If you have any idea about what kind of software we should try, please =
let me know. You've helped me enough already!
>=20
> Thank you and kind regards,
> Dejan Markic
> ________________________________________
> From: Flavio Junqueira [fpjunqueira@yahoo.com]
> Sent: Thursday, April 16, 2015 10:29 PM
> To: Dejan Markic
> Cc: user@zookeeper.apache.org
> Subject: Re: Transaction logs and snapshots
>=20
> Another think you could do is to make snapCount very large so that =
snapshots are created infrequently. But, let me step back and ask you =
why you think ZK is a good fit for your project. It isn't clear to me =
that your case is a good one for ZK.
>=20
> -Flavio
>=20
>=20
>> On 16 Apr 2015, at 11:01, Dejan Markic <Dejan.Markic@mobik-ics.com> =
wrote:
>>=20
>> Hello!
>>=20
>> Log seems to be always 67.108.880 bytes.
>> Snapshots are currently between 30-40MB. Snapshot is created almost =
every minute.
>> Yes, data durability is not important at all. Once the session ends =
(it may last between 0 and few minutes, average around 1-2 minutes =
maybe), I don't need it anymore. I regulary remove  nodes that are not =
changed for more than 10 minutes.
>> I even recieve updates for sessions, so even if ZK looses data, I =
would get it back after few minutes.
>>=20
>> Thanks!
>>=20
>> Kind regards,
>> Dejan
>>=20
>>=20
>> -----Original Message-----
>> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> Sent: Thursday, April 16, 2015 11:49 AM
>> To: user@zookeeper.apache.org
>> Subject: Re: Transaction logs and snapshots
>>=20
>> Hi Dejan,
>> For a typical ZK application, granularity of hours is more than =
enough, since it is supposed to be an infrequent background task. In =
your case, it sounds like durability isn't an important property because =
if it is you shouldn't be getting rid of disk data this fast. I'm also =
wondering about the amount of data you're generating. What's the size of =
your snapshots and txn logs?
>> -Flavio
>>=20
>>=20
>>    On Thursday, April 16, 2015 10:26 AM, Dejan Markic =
<Dejan.Markic@mobik-ics.com> wrote:
>>=20
>>=20
>>=20
>> Hello Flavio!
>>=20
>> Would that mean, that zkCleanup.sh would not be needed?
>> PurgeInterval is minimum 1 hour? Why is it so high?
>>=20
>> Thanks!
>>=20
>> Kind regards,
>> Dejan Markic
>>=20
>>=20
>> -----Original Message-----
>> From: Flavio Junqueira [mailto:fpjunqueira@yahoo.com.INVALID]
>> Sent: Thursday, April 16, 2015 11:15 AM
>> To: user@zookeeper.apache.org
>> Subject: Re: Transaction logs and snapshots
>>=20
>> Hi Dejan,
>> Check if the autopurge feature solves your problem:
>> =
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_advancedConf=
iguration
>>=20
>> -Flavio
>>=20
>>=20
>>    On Thursday, April 16, 2015 9:17 AM, Dejan Markic =
<Dejan.Markic@mobik-ics.com> wrote:
>>=20
>>=20
>>=20
>> Hello all!
>>=20
>> We are running 3 ZK servers in ensemble, and ZK is processing a lot =
of commands per seconds. There are probably around 300 nodes =
created/checked/set/get per second.
>> Since we have only information about live sessions we handle in ZK, =
we don't need any data persistency - eg: we can stop all nodes, clean =
all transaction logs/snapshots, and start them up again, without any =
issues.
>> Since we have a lot of requests/changes, we have moved dataDir onto =
ramdisk, so we have no problems with disk IOPS, etc.
>> Is there a way, to minimze the usage of snapshots/logs so ramdisk =
would not get filled up? It happens that transaction logs/snapshots grow =
so large, that we run out of space on ramdisk.
>> We issue >/usr/share/zookeeper/bin/zkCleanup.sh -n 3< every 2 =
minutes, so this should cleanup the dataDir quite often. Why is >count =
number of snapshots/logs to keep< limited to 3 and not below?
>> I assume, in my setup, I don't even need snapshots/logs to be stored =
after they are not actively needed?
>> So my basic questions are:
>> - can I somehow get rid of snapshot/logs sooner, more often ... ?
>> - when is snapshot created? Can it be created sooner, so it would be =
smaller?
>> - Is it possible to get rid of snapshot/logs all together?
>>=20
>> Thank you for all your inputs and kind regards, Dejan Markic
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>=20