What about the following solution:

Create cache: IgniteCache<Integer, Event>, where key is growing integer.
Asssign keys using IgniteAtomicSequence [1]
Listen for cache put events
When put is done and event group id is "next", process all entries from cache where id < event.key

[1] https://apacheignite.readme.io/docs/id-generator

2016-06-05 15:19 GMT+03:00 zshamrock <aliaksandr.kazlou@gmail.com>:
Are there features in Ignite which would support running an infinite (while
the cluster is up and running) job? For example, continuously reading values
from the distributed queue? So to implement producer/consumer pattern, where
there could be multiple producers, but I want to limit number of consumers,
ideally per specific key/group or if it is not possible, just to have one
consumer per queue.

If I asynchronously submit affinity ignite job with `queue.affinityRun()`
what is the implication of the this job never to finish? Will it consume the
thread from the ExecutorService thread pool on the running node forever

To give a better a context, this is the problem I am trying to solve (maybe
there are even other approaches to  solve it, and I am looking into the
completely wrong direction?):
- there are application events coming periodically (based on the application
state changes)
- I have to accumulate these events until the block of the events is
"complete" (completion is defined by an application rule), as until the
group is complete nothing can be done/processed
- when the group is complete I have to process all of the events in the
group (as one complete chunk), while still accepting new events coming for
now another "incomplete" group
- and repeat since the beginning

So, far I came with the following solution:
- collect and keep all the events in the distributed IgniteQueue
- when the application notifies the completion of the group, I trigger
`queue.affinityRun()` (as I have to do a peek before removing the event from
the queue, so I want to run the execution logic on the node where the queue
is stored, they are small and run in collocated mode, and so peek will not
do an unnecessary network call)
[the reason for a peek, is that even if I receive the application event of
the group completion, due to the way events are stored (in the queue), I
don't know where the group ends, only where it starts (head of the queue),
but looking into the event itself, I can detect whether it is still from the
same group, or already from a new incomplete group, this is why I have to do
peek, as if I do poll/take first then I have to the put the element back
into the head of the queue (which obviously is not possible, as it is a
queue and not a dequeue), then I have to store this element/event somewhere
else, and on the next job submitted start with this stored event as a "head"
of the queue, and only then switch back to the real queue. As I don't want
this extra complexity, I am ready to pay a price for an extra peek before
the take]
- implement custom CollisionSpi which will understand whether there is
already a running job for the given queue, and if so, keeps the newly
submitted job in the waiting list
[here again due to the fact how events are stored (in the queue) I don't
allow multiple jobs running against same queue at the same time, as taking
the element from the middle of one group already processing group is
obviously an error, so I have to limit (to 1) the number of parallel jobs
against the given queue]
- it also requires to submit a new ignite job (distributed closure) on the
queue every time the application triggers/generates a completion group
event, which requires/should schedule a queue processing (also see above on
the overall number of the simultaneous jobs)

I thought about other alternative solutions, but all of them turned out to
be more complex, and involve more moving parts (as for example, for the
distributed queue Ignite manages atomicity, and consistency, with other
approaches I have to do it all manually, which I just want to minimize) and
more logic to maintain and ensure correctness.

Is there any other suitable alternative for the problem described above?

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Running-an-infinite-job-use-case-inside-or-alternatives-tp5430.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Best regards,
Alexei Scherbakov