ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Scherbakov <alexey.scherbak...@gmail.com>
Subject Re: Running an infinite job? (use case inside) or alternatives
Date Mon, 06 Jun 2016 16:33:01 GMT

What about the following solution:

Create cache: IgniteCache<Integer, Event>, where key is growing integer.
Asssign keys using IgniteAtomicSequence [1]
Listen for cache put events
When put is done and event group id is "next", process all entries from
cache where id < event.key

[1] https://apacheignite.readme.io/docs/id-generator

2016-06-05 15:19 GMT+03:00 zshamrock <aliaksandr.kazlou@gmail.com>:

> Are there features in Ignite which would support running an infinite (while
> the cluster is up and running) job? For example, continuously reading
> values
> from the distributed queue? So to implement producer/consumer pattern,
> where
> there could be multiple producers, but I want to limit number of consumers,
> ideally per specific key/group or if it is not possible, just to have one
> consumer per queue.
> If I asynchronously submit affinity ignite job with `queue.affinityRun()`
> what is the implication of the this job never to finish? Will it consume
> the
> thread from the ExecutorService thread pool on the running node forever
> then?
> To give a better a context, this is the problem I am trying to solve (maybe
> there are even other approaches to  solve it, and I am looking into the
> completely wrong direction?):
> - there are application events coming periodically (based on the
> application
> state changes)
> - I have to accumulate these events until the block of the events is
> "complete" (completion is defined by an application rule), as until the
> group is complete nothing can be done/processed
> - when the group is complete I have to process all of the events in the
> group (as one complete chunk), while still accepting new events coming for
> now another "incomplete" group
> - and repeat since the beginning
> So, far I came with the following solution:
> - collect and keep all the events in the distributed IgniteQueue
> - when the application notifies the completion of the group, I trigger
> `queue.affinityRun()` (as I have to do a peek before removing the event
> from
> the queue, so I want to run the execution logic on the node where the queue
> is stored, they are small and run in collocated mode, and so peek will not
> do an unnecessary network call)
> [the reason for a peek, is that even if I receive the application event of
> the group completion, due to the way events are stored (in the queue), I
> don't know where the group ends, only where it starts (head of the queue),
> but looking into the event itself, I can detect whether it is still from
> the
> same group, or already from a new incomplete group, this is why I have to
> do
> peek, as if I do poll/take first then I have to the put the element back
> into the head of the queue (which obviously is not possible, as it is a
> queue and not a dequeue), then I have to store this element/event somewhere
> else, and on the next job submitted start with this stored event as a
> "head"
> of the queue, and only then switch back to the real queue. As I don't want
> this extra complexity, I am ready to pay a price for an extra peek before
> the take]
> - implement custom CollisionSpi which will understand whether there is
> already a running job for the given queue, and if so, keeps the newly
> submitted job in the waiting list
> [here again due to the fact how events are stored (in the queue) I don't
> allow multiple jobs running against same queue at the same time, as taking
> the element from the middle of one group already processing group is
> obviously an error, so I have to limit (to 1) the number of parallel jobs
> against the given queue]
> - it also requires to submit a new ignite job (distributed closure) on the
> queue every time the application triggers/generates a completion group
> event, which requires/should schedule a queue processing (also see above on
> the overall number of the simultaneous jobs)
> I thought about other alternative solutions, but all of them turned out to
> be more complex, and involve more moving parts (as for example, for the
> distributed queue Ignite manages atomicity, and consistency, with other
> approaches I have to do it all manually, which I just want to minimize) and
> more logic to maintain and ensure correctness.
> Is there any other suitable alternative for the problem described above?
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Running-an-infinite-job-use-case-inside-or-alternatives-tp5430.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Best regards,
Alexei Scherbakov

View raw message