ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Semyon Boikov <sboi...@gridgain.com>
Subject Re: Continuous queries changes
Date Thu, 13 Aug 2015 11:48:06 GMT
After more thinking about ordering on backups instead of
thread-per-partition I believe this approach won't work.

First it is possible that primary executes for some key <update 1>, <update
2> (from different threads), then crashes and only <update 1> message is
sent to backup, so backup won't be able to restore correct ordering.

Also if the same partition can be updated concurrently it is not possible
to associate per-partition update counter with each update so it is not
clear how to deal with duplicated messages and backup acknowledgement.

But since there is no thread-per-partition mode for transactional cache do
these issues affect transactional cache as well?


On Wed, Aug 12, 2015 at 4:49 PM, Semyon Boikov <sboikov@gridgain.com> wrote:

> Hi,
>
> I started to work on ignite-426 and see that current implementation of
> thread-per-partition approach required for this feature has serious impact
> on performance.
>
> As I understand thread-per-partition is required to avoid issue with
> update order on backup nodes: now it is possible that on primary node
> updates were executed 'update 1', 'update 2' and on backup nodes update
> messages can be handled like <update 2>, <update 1>.
>
> But now backup will eventually have correct value due to the fact that
> cache entry version grows, so 'update 2' will always have version greater
> than 'update 1'.
>
> What if on backup we will sort continuous query events by cache update
> version? In this case when backup queue is flushed messages will be
> correctly ordered and thread-per-partition mode is not needed.
>
> Do you see any problems with this approach, does it worth to try this?
>
> Thanks
>
>
> On Thu, Jul 23, 2015 at 8:53 AM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Igniters,
>>
>> Based on discussions with our users I came to conclusion that our
>> continuous query implementation is not good enough for use cases with
>> strong consistency requirements, because there is a possibility to lose
>> updates in case of topology change.
>>
>> So I started working on https://issues.apache.org/jira/browse/IGNITE-426
>> and I hope to finish in in couple of weeks so that we can include it in
>> next release.
>>
>> I have the following design in mind:
>>
>>    - Maintain updates queue on backup node(s) in addition to primary node.
>>    - If primary node crushes, this queue is flushed to listening clients.
>>    - To avoid notification duplicates we will have a per-partition update
>>    counter. Once an entry in some partition is updated, counter for this
>>    partition is incremented on both primary and backups. The value of this
>>    counter is also sent along with the update to the client, which also
>>    maintains the copy of this mapping. If at some moment it receives an
>> update
>>    with the counter less than in its local map, this update is a
>> duplicate and
>>    can be discarded.
>>    - Also need to figure out the best way to clean the backup queue if
>>    topology is stable. Will be happy to hear any suggestions :)
>>
>> To make all this work we also need to implement thread-per-partition mode
>> in atomic cache, because now updates order on backup nodes can differ from
>> the primary node: https://issues.apache.org/jira/browse/IGNITE-104. I'm
>> already working on this.
>>
>> Feel free to share your thoughts!
>>
>> -Val
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message