ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Continuous queries changes
Date Sat, 25 Jul 2015 06:40:15 GMT
Andrey,

I mean the queue of update events that is collected on backup nodes and
flushed to listening clients in case of topology change.

-Val

On Fri, Jul 24, 2015 at 3:16 PM, Andrey Kornev <andrewkornev@hotmail.com>
wrote:

> Val,
>
> Could you please elaborate what you mean by "updates queue" you plan to
> maintain on the primary and backup nodes?
>
> Thanks
> Andrey
>
> > Date: Fri, 24 Jul 2015 17:51:48 +0300
> > Subject: Re: Continuous queries changes
> > From: yzhdanov@apache.org
> > To: dev@ignite.incubator.apache.org
> >
> > Val,
> >
> > I have idea on how to clean up backup queue.
> >
> > 1. Our communication uses acks. So, you can determine [on server node]
> > whether client received the update from local server or not. I think you
> > can easily change existing code to get notifications on ack receiving
> (this
> > way you dont need to introduce your own acks).
> > 2. How do you know when evict from backup? Each message that client acks
> > corresponds to some per-partition long value you talked above (great
> idea,
> > btw!). Servers can exchange per-partition long value that corresponds to
> > the latest acked message and that's the way how backups can safely evict
> > from the queue.
> >
> > Let me know if you have questions.
> >
> > --Yakov
> >
> > 2015-07-23 8:53 GMT+03:00 Valentin Kulichenko <
> valentin.kulichenko@gmail.com
> > >:
> >
> > > Igniters,
> > >
> > > Based on discussions with our users I came to conclusion that our
> > > continuous query implementation is not good enough for use cases with
> > > strong consistency requirements, because there is a possibility to lose
> > > updates in case of topology change.
> > >
> > > So I started working on
> https://issues.apache.org/jira/browse/IGNITE-426
> > > and I hope to finish in in couple of weeks so that we can include it in
> > > next release.
> > >
> > > I have the following design in mind:
> > >
> > >    - Maintain updates queue on backup node(s) in addition to primary
> node.
> > >    - If primary node crushes, this queue is flushed to listening
> clients.
> > >    - To avoid notification duplicates we will have a per-partition
> update
> > >    counter. Once an entry in some partition is updated, counter for
> this
> > >    partition is incremented on both primary and backups. The value of
> this
> > >    counter is also sent along with the update to the client, which also
> > >    maintains the copy of this mapping. If at some moment it receives an
> > > update
> > >    with the counter less than in its local map, this update is a
> duplicate
> > > and
> > >    can be discarded.
> > >    - Also need to figure out the best way to clean the backup queue if
> > >    topology is stable. Will be happy to hear any suggestions :)
> > >
> > > To make all this work we also need to implement thread-per-partition
> mode
> > > in atomic cache, because now updates order on backup nodes can differ
> from
> > > the primary node: https://issues.apache.org/jira/browse/IGNITE-104.
> I'm
> > > already working on this.
> > >
> > > Feel free to share your thoughts!
> > >
> > > -Val
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message