zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ibrahim El-sanosi <ibrahimsaba...@gmail.com>
Subject Re: Dealing with old delivered requests in memory
Date Tue, 22 Mar 2016 14:53:25 GMT
Thank you Patrick,



>That's correct. ZK is an in-memory store. As such all znodes

>representing the current state are held in memory.

>

>>  2. If the leader crashes or lost a quorum, recovery mechanism applies

>> latest snapshots. In this scenario, it is possible to eliminate the old
ZK

>> states and could save memory space.

>>

>

>This won't affect the in-memory state. It will be the same as prior to

>the recovery.



Observation (2), I assume the worst scenario when all ZK servers want down
and start recovery phase (No ZK states persent in memory at the moment).
Does ZK servers reapply all snapshot in memory or just takes the latest one?



Ibrahim

On Tue, Mar 22, 2016 at 2:43 PM, Patrick Hunt <phunt@apache.org> wrote:

> On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi
> <ibrahimsabattt@gmail.com> wrote:
> > Ok, Thank you for answering the question.
> >
> >  So, what I understand from this thread is as following:
> >
> >  1. When the crash does occur (Leader does not crash and majority of
> > servers is always up), the state from Time T0 (ZK start of ZK) and until
> > time Tn (ZK state of now) will be in memory (even the size of ZK state
> > reaches to the memory size).
> >
>
> That's correct. ZK is an in-memory store. As such all znodes
> representing the current state are held in memory.
>
> >  2. If the leader crashes or lost a quorum, recovery mechanism applies
> > latest snapshots. In this scenario, it is possible to eliminate the old
> ZK
> > states and could save memory space.
> >
>
> This won't affect the in-memory state. It will be the same as prior to
> the recovery.
>
> Patrick
>
> >
> >  Is that right?
> >
> >
> >
> > Regards,
> >
> >  Ibrahim
> >
> > On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <fpj@apache.org>
> wrote:
> >
> >> It is currently as Jordan says, although it is not entirely unreasonable
> >> to think of memory as cache and keep in memory only the current working
> >> set, possibly using say an SSD to store the remaining part of the state
> >> that you don't want to keep in memory. One key issue here is that if you
> >> have a pretty large state (that you don't want to keep all in memory),
> then
> >> you'll end up increasing worst-case recovery time.
> >>
> >> We certainly do not have any kind of expiration mechanism for znodes
> other
> >> than the ephemeral flag, which is associated to sessions.
> >>
> >> -Flavio
> >>
> >> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi <ibrahimsabattt@gmail.com
> >
> >> wrote:
> >> >
> >> > Thank you for replaying.
> >> >
> >> >
> >> > Take this example, Zookeeper started on 1/1/2015, and assume ZK
> servers
> >> are
> >> > never crashed. Also, the ZK are very busy, reciveing continusly write
> >> > requests from clients and accordingly snapshots are generated
> overtime.
> >> On
> >> > 1/3/2015, what  Znodes will be in memory data tree? do all Znodes
> still
> >> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed
> occur.
> >> > Or Dsnapshhot is taken the data tree is
> >> >
> >> >
> >> > Note that  I am not asking about data in log or snapshoot. I am asking
> >> > about the current data in memory.
> >> >
> >> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <fpj@apache.org>
> >> wrote:
> >> >
> >> >> Hi Ibrahim,
> >> >>
> >> >> Are you asking about how we compact old logs? We do it by taking
> >> snapshots
> >> >> so that upon recovery, we only load the latest snapshot and replay
> the
> >> txn
> >> >> log from the snapshot tag. The snapshot tag is the last zxid
> committed
> >> when
> >> >> we start producing the snapshot.
> >> >>
> >> >> We don't actually delete anything, though, unless you intentionally
> turn
> >> >> on auto purge:
> >> >>
> >> >>
> >>
> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
> >> >> <
> >> >>
> >>
> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
> >> >>>
> >> >>
> >> >> -Flavio
> >> >>
> >> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi <
> ibrahimsabattt@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Hi all,
> >> >>>
> >> >>> Assume the Zookeeper have been running for about one year (from
> >> 1/1/2015
> >> >>> until now), how does ZooKeeper deal with old delivered write
> requests
> >> (To
> >> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015). I am
> >> assuming
> >> >>> the old delivered request are no longer used.
> >> >>>
> >> >>> On the other words, what strategy ZK does to find and delete unused
> >> >> Znodes?
> >> >>>
> >> >>> Ibrahim
> >> >>
> >> >>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message