zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: Dealing with old delivered requests in memory
Date Wed, 23 Mar 2016 16:44:44 GMT
On Tue, Mar 22, 2016 at 7:53 AM, ibrahim El-sanosi
<ibrahimsabattt@gmail.com> wrote:
> Thank you Patrick,
>
>
>
>>That's correct. ZK is an in-memory store. As such all znodes
>
>>representing the current state are held in memory.
>
>>
>
>>>  2. If the leader crashes or lost a quorum, recovery mechanism applies
>
>>> latest snapshots. In this scenario, it is possible to eliminate the old
> ZK
>
>>> states and could save memory space.
>
>>>
>
>>
>
>>This won't affect the in-memory state. It will be the same as prior to
>
>>the recovery.
>
>
>
> Observation (2), I assume the worst scenario when all ZK servers want down
> and start recovery phase (No ZK states persent in memory at the moment).
> Does ZK servers reapply all snapshot in memory or just takes the latest one?
>

The most recent valid snapshot is read and any transactions written to
the logs subsequent to that snapshot are replayed.

Patrick

>
>
> Ibrahim
>
> On Tue, Mar 22, 2016 at 2:43 PM, Patrick Hunt <phunt@apache.org> wrote:
>
>> On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi
>> <ibrahimsabattt@gmail.com> wrote:
>> > Ok, Thank you for answering the question.
>> >
>> >  So, what I understand from this thread is as following:
>> >
>> >  1. When the crash does occur (Leader does not crash and majority of
>> > servers is always up), the state from Time T0 (ZK start of ZK) and until
>> > time Tn (ZK state of now) will be in memory (even the size of ZK state
>> > reaches to the memory size).
>> >
>>
>> That's correct. ZK is an in-memory store. As such all znodes
>> representing the current state are held in memory.
>>
>> >  2. If the leader crashes or lost a quorum, recovery mechanism applies
>> > latest snapshots. In this scenario, it is possible to eliminate the old
>> ZK
>> > states and could save memory space.
>> >
>>
>> This won't affect the in-memory state. It will be the same as prior to
>> the recovery.
>>
>> Patrick
>>
>> >
>> >  Is that right?
>> >
>> >
>> >
>> > Regards,
>> >
>> >  Ibrahim
>> >
>> > On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <fpj@apache.org>
>> wrote:
>> >
>> >> It is currently as Jordan says, although it is not entirely unreasonable
>> >> to think of memory as cache and keep in memory only the current working
>> >> set, possibly using say an SSD to store the remaining part of the state
>> >> that you don't want to keep in memory. One key issue here is that if you
>> >> have a pretty large state (that you don't want to keep all in memory),
>> then
>> >> you'll end up increasing worst-case recovery time.
>> >>
>> >> We certainly do not have any kind of expiration mechanism for znodes
>> other
>> >> than the ephemeral flag, which is associated to sessions.
>> >>
>> >> -Flavio
>> >>
>> >> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi <ibrahimsabattt@gmail.com
>> >
>> >> wrote:
>> >> >
>> >> > Thank you for replaying.
>> >> >
>> >> >
>> >> > Take this example, Zookeeper started on 1/1/2015, and assume ZK
>> servers
>> >> are
>> >> > never crashed. Also, the ZK are very busy, reciveing continusly write
>> >> > requests from clients and accordingly snapshots are generated
>> overtime.
>> >> On
>> >> > 1/3/2015, what  Znodes will be in memory data tree? do all Znodes
>> still
>> >> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed
>> occur.
>> >> > Or Dsnapshhot is taken the data tree is
>> >> >
>> >> >
>> >> > Note that  I am not asking about data in log or snapshoot. I am asking
>> >> > about the current data in memory.
>> >> >
>> >> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <fpj@apache.org>
>> >> wrote:
>> >> >
>> >> >> Hi Ibrahim,
>> >> >>
>> >> >> Are you asking about how we compact old logs? We do it by taking
>> >> snapshots
>> >> >> so that upon recovery, we only load the latest snapshot and replay
>> the
>> >> txn
>> >> >> log from the snapshot tag. The snapshot tag is the last zxid
>> committed
>> >> when
>> >> >> we start producing the snapshot.
>> >> >>
>> >> >> We don't actually delete anything, though, unless you intentionally
>> turn
>> >> >> on auto purge:
>> >> >>
>> >> >>
>> >>
>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>> >> >> <
>> >> >>
>> >>
>> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
>> >> >>>
>> >> >>
>> >> >> -Flavio
>> >> >>
>> >> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi <
>> ibrahimsabattt@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Hi all,
>> >> >>>
>> >> >>> Assume the Zookeeper have been running for about one year (from
>> >> 1/1/2015
>> >> >>> until now), how does ZooKeeper deal with old delivered write
>> requests
>> >> (To
>> >> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015).
I am
>> >> assuming
>> >> >>> the old delivered request are no longer used.
>> >> >>>
>> >> >>> On the other words, what strategy ZK does to find and delete
unused
>> >> >> Znodes?
>> >> >>>
>> >> >>> Ibrahim
>> >> >>
>> >> >>
>> >>
>> >>
>>

Mime
View raw message