activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Message group checkpoints...(processing N messages with one task).
Date Sun, 24 Jan 2016 20:16:45 GMT
I have a pattern which I think I need advice on...

I have three tasks... each a type of message consumer.

Let's call them A, B, an C.

A runs once,, creates 15 messages, sends them to B... then B process these
messages then generates 15 new messages.

However, they need to be combined into a group, and sent to C all at once,
in one composite message.

So its similar to map/reduce in a way in that C should execute once with a
block of these 15 messages.

Conceptually I'm calling them (message group checkpoints).. but I'm
wondering if there's already a more formal name for this concept.

I'm not sure the best way to handle this with ActiveMQ.

One strategy is that I could have one queue per C tasks (the final tasks)
and then have C running and consuming them one at a time, and then
performing the execution (and message commit) once it receives all 15
messages.

I HAVE to get all the messages until I can make a decision, I can't stream
process them unfortunately because the algorithm needs all data points.

I could use a database.. but the problem there is that it would incur some
database and Cassandra (in our case) doesn't handle this queue pattern very
well.

Another idea is to use a series of queues ... say aggregation_0,
aggregation_1, aggregation_2,...

then I receive these messages into the first queue (aggregation_0), then
sort it, if any of the groups are finished I send them on to the final
destination task.  If any are unfinished then I overflow them on to
aggregation1 (pre-sorted)...

Thoughts?

-- 

We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message