Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAOafRRdoa9d6dd-SUW1mC0_i=0h_mEon84-LwPq3=GHxX5QO-w@mail.gmail.com>
References: 
 <CAOafRReRFC7bmY-=zKfnU9sYDd9w5bZmqVpV8irdTdjZQ9Lcgw@mail.gmail.com>
 <BBCD5C0F-DD22-44C8-B21D-B31D7637D86C@apache.org>
 <CAOafRRdbw_hBNzi8xZYVE2WTAaU0+M7LgXZJCr+c_kPrL7Gv+g@mail.gmail.com>
 <A09B3B47-DE36-4E83-BAB3-E209ED60B26C@apache.org>
 <CAOafRRdoa9d6dd-SUW1mC0_i=0h_mEon84-LwPq3=GHxX5QO-w@mail.gmail.com>
From: Patrick Hunt <phunt@apache.org>
Date: Tue, 22 Mar 2016 07:43:25 -0700
Message-ID: 
 <CANLc_9Lv5McOEY+sb1N2jXWXkTtDEz38GwiR3gwb1XgDXsExRA@mail.gmail.com>
Subject: Re: Dealing with old delivered requests in memory
To: UserZooKeeper <user@zookeeper.apache.org>
Content-Type: text/plain; charset=UTF-8

On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi
<ibrahimsabattt@gmail.com> wrote:
> Ok, Thank you for answering the question.
>
>  So, what I understand from this thread is as following:
>
>  1. When the crash does occur (Leader does not crash and majority of
> servers is always up), the state from Time T0 (ZK start of ZK) and until
> time Tn (ZK state of now) will be in memory (even the size of ZK state
> reaches to the memory size).
>

That's correct. ZK is an in-memory store. As such all znodes
representing the current state are held in memory.

>  2. If the leader crashes or lost a quorum, recovery mechanism applies
> latest snapshots. In this scenario, it is possible to eliminate the old ZK
> states and could save memory space.
>

This won't affect the in-memory state. It will be the same as prior to
the recovery.

Patrick

>
>  Is that right?
>
>
>
> Regards,
>
>  Ibrahim
>
> On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <fpj@apache.org> wrote:
>
>> It is currently as Jordan says, although it is not entirely unreasonable
>> to think of memory as cache and keep in memory only the current working
>> set, possibly using say an SSD to store the remaining part of the state
>> that you don't want to keep in memory. One key issue here is that if you
>> have a pretty large state (that you don't want to keep all in memory), then
>> you'll end up increasing worst-case recovery time.
>>
>> We certainly do not have any kind of expiration mechanism for znodes other
>> than the ephemeral flag, which is associated to sessions.
>>
>> -Flavio
>>
>> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi <ibrahimsabattt@gmail.com>
>> wrote:
>> >
>> > Thank you for replaying.
>> >
>> >
>> > Take this example, Zookeeper started on 1/1/2015, and assume ZK servers
>> are
>> > never crashed. Also, the ZK are very busy, reciveing continusly write
>> > requests from clients and accordingly snapshots are generated overtime.
>> On
>> > 1/3/2015, what  Znodes will be in memory data tree? do all Znodes still
>> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed occur.
>> > Or Dsnapshhot is taken the data tree is
>> >
>> >
>> > Note that  I am not asking about data in log or snapshoot. I am asking
>> > about the current data in memory.
>> >
>> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <fpj@apache.org>
>> wrote:
>> >
>> >> Hi Ibrahim,
>> >>
>> >> Are you asking about how we compact old logs? We do it by taking
>> snapshots
>> >> so that upon recovery, we only load the latest snapshot and replay the
>> txn
>> >> log from the snapshot tag. The snapshot tag is the last zxid committed
>> when
>> >> we start producing the snapshot.
>> >>
>> >> We don't actually delete anything, though, unless you intentionally turn
>> >> on auto purge:
>> >>
>> >>
>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>> >> <
>> >>
>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>> >>>
>> >>
>> >> -Flavio
>> >>
>> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi <ibrahimsabattt@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Assume the Zookeeper have been running for about one year (from
>> 1/1/2015
>> >>> until now), how does ZooKeeper deal with old delivered write requests
>> (To
>> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015). I am
>> assuming
>> >>> the old delivered request are no longer used.
>> >>>
>> >>> On the other words, what strategy ZK does to find and delete unused
>> >> Znodes?
>> >>>
>> >>> Ibrahim
>> >>
>> >>
>>
>>