ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Wilson <raymond_wil...@trimble.com>
Subject Re: Using a cache as an affinity co-located processing buffer in Ignite.Net
Date Thu, 26 Apr 2018 18:22:31 GMT
Val,

What I really want is a .Net implementation of the Ignite distributed queue, assuming co-located
server processing can pull members off the queue that are co-located with that node (yeah,
that's probably mangling queue semantics a bit, but not that badly :) )

I have previously made a POC code implementation where a service is deployed to every node
and runs a continuous query against the local members of the cache (this one does not abuse
remote filters). Ie: one CQ per node. This approach requires a single context to issue the
service deployment (which goes back to the best practices question)

I have a second implementation where a single context issues the CQ and relies on remote filters
to deal with the cache entries. To be fair, the remote filter just shunts the item into another
class which asynchronously processes them (so I don't think there is a significant locking
issue), though this introduces the need for a static singleton to handle the dual remote filter
contexts for the scan query and continuous query aspects of the CQ.

The exactly once guarantee differences are food for thought. The processing is idempotent,
but still not desirable to process things more than once if not needed. 

Sent from my iPhone

> On 27/04/2018, at 5:49 AM, vkulichenko <valentin.kulichenko@gmail.com> wrote:
> 
> Raymond,
> 
> If you want to have a single continuous query on one of the server nodes,
> then I think singleton service is the best option to achieve that. It will
> not only guarantee that there is one query at a time, but will also make
> sure to redeploy it in case of failure.
> 
> Also I would be accurate with doing processing in the remote filter. Filter
> is not designed for this, it's designed to do the filtering of updates
> before sending them to listening node. Major pitfall here is that it's
> invoked within entry lock, so any heavy processing would block the entry.
> Moreover, synchronous cache or other operations can cause deadlocks and/or
> thread starvation. Finally, keep in mind that filter can be invoked multiple
> times (at least for primary and backups, sometimes even more in case of
> failures). There is no exactly-once guarantee for remote filter, it's only
> applied to local listener.
> 
> -Val
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message