zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Zimmerman <jor...@jordanzimmerman.com>
Subject Re: Dealing with old delivered requests in memory
Date Fri, 18 Mar 2016 16:59:44 GMT
Also, since 3.5, there are “container” nodes that auto-delete once emptied of children.

-JZ

> On Mar 18, 2016, at 11:49 AM, Flavio Junqueira <fpj@apache.org> wrote:
> 
> It is currently as Jordan says, although it is not entirely unreasonable to think of
memory as cache and keep in memory only the current working set, possibly using say an SSD
to store the remaining part of the state that you don't want to keep in memory. One key issue
here is that if you have a pretty large state (that you don't want to keep all in memory),
then you'll end up increasing worst-case recovery time.
> 
> We certainly do not have any kind of expiration mechanism for znodes other than the ephemeral
flag, which is associated to sessions.
> 
> -Flavio
> 
>> On 18 Mar 2016, at 16:38, ibrahim El-sanosi <ibrahimsabattt@gmail.com> wrote:
>> 
>> Thank you for replaying.
>> 
>> 
>> Take this example, Zookeeper started on 1/1/2015, and assume ZK servers are
>> never crashed. Also, the ZK are very busy, reciveing continusly write
>> requests from clients and accordingly snapshots are generated overtime. On
>> 1/3/2015, what  Znodes will be in memory data tree? do all Znodes still
>> store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed occur.
>> Or Dsnapshhot is taken the data tree is
>> 
>> 
>> Note that  I am not asking about data in log or snapshoot. I am asking
>> about the current data in memory.
>> 
>> On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <fpj@apache.org> wrote:
>> 
>>> Hi Ibrahim,
>>> 
>>> Are you asking about how we compact old logs? We do it by taking snapshots
>>> so that upon recovery, we only load the latest snapshot and replay the txn
>>> log from the snapshot tag. The snapshot tag is the last zxid committed when
>>> we start producing the snapshot.
>>> 
>>> We don't actually delete anything, though, unless you intentionally turn
>>> on auto purge:
>>> 
>>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>>> <
>>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>>>> 
>>> 
>>> -Flavio
>>> 
>>>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi <ibrahimsabattt@gmail.com>
>>> wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> Assume the Zookeeper have been running for about one year (from 1/1/2015
>>>> until now), how does ZooKeeper deal with old delivered write requests (To
>>>> optimize a memory used) (say from 1/1/2015 to 03/04/2015). I am assuming
>>>> the old delivered request are no longer used.
>>>> 
>>>> On the other words, what strategy ZK does to find and delete unused
>>> Znodes?
>>>> 
>>>> Ibrahim
>>> 
>>> 
> 


Mime
View raw message