zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ibrahim El-sanosi <ibrahimsaba...@gmail.com>
Subject Re: Dealing with old delivered requests in memory
Date Wed, 23 Mar 2016 16:56:59 GMT
Thank you Patrick,


Ibrahim

On Wed, Mar 23, 2016 at 4:44 PM, Patrick Hunt <phunt@apache.org> wrote:

> On Tue, Mar 22, 2016 at 7:53 AM, ibrahim El-sanosi
> <ibrahimsabattt@gmail.com> wrote:
> > Thank you Patrick,
> >
> >
> >
> >>That's correct. ZK is an in-memory store. As such all znodes
> >
> >>representing the current state are held in memory.
> >
> >>
> >
> >>>  2. If the leader crashes or lost a quorum, recovery mechanism applies
> >
> >>> latest snapshots. In this scenario, it is possible to eliminate the old
> > ZK
> >
> >>> states and could save memory space.
> >
> >>>
> >
> >>
> >
> >>This won't affect the in-memory state. It will be the same as prior to
> >
> >>the recovery.
> >
> >
> >
> > Observation (2), I assume the worst scenario when all ZK servers want
> down
> > and start recovery phase (No ZK states persent in memory at the moment).
> > Does ZK servers reapply all snapshot in memory or just takes the latest
> one?
> >
>
> The most recent valid snapshot is read and any transactions written to
> the logs subsequent to that snapshot are replayed.
>
> Patrick
>
> >
> >
> > Ibrahim
> >
> > On Tue, Mar 22, 2016 at 2:43 PM, Patrick Hunt <phunt@apache.org> wrote:
> >
> >> On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi
> >> <ibrahimsabattt@gmail.com> wrote:
> >> > Ok, Thank you for answering the question.
> >> >
> >> >  So, what I understand from this thread is as following:
> >> >
> >> >  1. When the crash does occur (Leader does not crash and majority of
> >> > servers is always up), the state from Time T0 (ZK start of ZK) and
> until
> >> > time Tn (ZK state of now) will be in memory (even the size of ZK state
> >> > reaches to the memory size).
> >> >
> >>
> >> That's correct. ZK is an in-memory store. As such all znodes
> >> representing the current state are held in memory.
> >>
> >> >  2. If the leader crashes or lost a quorum, recovery mechanism applies
> >> > latest snapshots. In this scenario, it is possible to eliminate the
> old
> >> ZK
> >> > states and could save memory space.
> >> >
> >>
> >> This won't affect the in-memory state. It will be the same as prior to
> >> the recovery.
> >>
> >> Patrick
> >>
> >> >
> >> >  Is that right?
> >> >
> >> >
> >> >
> >> > Regards,
> >> >
> >> >  Ibrahim
> >> >
> >> > On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <fpj@apache.org>
> >> wrote:
> >> >
> >> >> It is currently as Jordan says, although it is not entirely
> unreasonable
> >> >> to think of memory as cache and keep in memory only the current
> working
> >> >> set, possibly using say an SSD to store the remaining part of the
> state
> >> >> that you don't want to keep in memory. One key issue here is that if
> you
> >> >> have a pretty large state (that you don't want to keep all in
> memory),
> >> then
> >> >> you'll end up increasing worst-case recovery time.
> >> >>
> >> >> We certainly do not have any kind of expiration mechanism for znodes
> >> other
> >> >> than the ephemeral flag, which is associated to sessions.
> >> >>
> >> >> -Flavio
> >> >>
> >> >> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi <
> ibrahimsabattt@gmail.com
> >> >
> >> >> wrote:
> >> >> >
> >> >> > Thank you for replaying.
> >> >> >
> >> >> >
> >> >> > Take this example, Zookeeper started on 1/1/2015, and assume ZK
> >> servers
> >> >> are
> >> >> > never crashed. Also, the ZK are very busy, reciveing continusly
> write
> >> >> > requests from clients and accordingly snapshots are generated
> >> overtime.
> >> >> On
> >> >> > 1/3/2015, what  Znodes will be in memory data tree? do all Znodes
> >> still
> >> >> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed
> >> occur.
> >> >> > Or Dsnapshhot is taken the data tree is
> >> >> >
> >> >> >
> >> >> > Note that  I am not asking about data in log or snapshoot. I am
> asking
> >> >> > about the current data in memory.
> >> >> >
> >> >> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <fpj@apache.org>
> >> >> wrote:
> >> >> >
> >> >> >> Hi Ibrahim,
> >> >> >>
> >> >> >> Are you asking about how we compact old logs? We do it by
taking
> >> >> snapshots
> >> >> >> so that upon recovery, we only load the latest snapshot and
replay
> >> the
> >> >> txn
> >> >> >> log from the snapshot tag. The snapshot tag is the last zxid
> >> committed
> >> >> when
> >> >> >> we start producing the snapshot.
> >> >> >>
> >> >> >> We don't actually delete anything, though, unless you
> intentionally
> >> turn
> >> >> >> on auto purge:
> >> >> >>
> >> >> >>
> >> >>
> >>
> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
> >> >> >> <
> >> >> >>
> >> >>
> >>
> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering
> >> >> >>>
> >> >> >>
> >> >> >> -Flavio
> >> >> >>
> >> >> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi <
> >> ibrahimsabattt@gmail.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Hi all,
> >> >> >>>
> >> >> >>> Assume the Zookeeper have been running for about one year
(from
> >> >> 1/1/2015
> >> >> >>> until now), how does ZooKeeper deal with old delivered
write
> >> requests
> >> >> (To
> >> >> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015).
I am
> >> >> assuming
> >> >> >>> the old delivered request are no longer used.
> >> >> >>>
> >> >> >>> On the other words, what strategy ZK does to find and
delete
> unused
> >> >> >> Znodes?
> >> >> >>>
> >> >> >>> Ibrahim
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message