activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drvillo <>
Subject Re: On message spooling
Date Mon, 15 Jan 2007 10:10:46 GMT

Hi James
thanks a lot for your feedback.

James.Strachan wrote:
> FWIW spooling to disk of transient messages is now in trunk to avoid
> running out of RAM when doing non-persistent messaging; I thought
> thats what the subject of this thread was gonna be about :)

Actually my messages are persistent, but I understand that journaled
persistence spool pendent messages when checkpointing, thus if 
I produce a big number of messages suddenly these are going to be
kept in memory for a while, right?
Anyway is there any testcase that shows how spooling works/can be triggered?

James.Strachan wrote:
> So if I understand you correctly, you want to receive a bunch of
> messages, store them somewhere then at a later date, when its a good
> time, send them right?
> If so I'd write a regular transactional consumer; consume the messages
> from A then, when you are ready, send them to their ultimate
> destination B and do a commit.
> If you are worried that queue A is too full or that there are messages
> on queue A which you want to 'move out of the way for a while' you
> could always move them to queue C; then forward on to queue B when you
> are ready. So using queue C as a temporary staging area for things
> that can wait.
> e.g. take all the messages off A as fast as you can, send the 'high
> priority' stuff to B, the low prority stuff to C; then when queues A
> and B are small enough, you can trickle feed from C back to B again
> (assuming that ordering between the high priority and low priority
> messages is not vital). Just use the MBean stats to watch the queue
> depths to know when you should consume more from C and move it to B.

This is exactly it.
The thing is, doing this by just using another destination ( C ) would 
need to keep these messages around in the broker anyway, wouldn't it?
What I have done is spooling manually messages using a kaha store
when the queue are in a certain state (i.e too full). This way messages
are actually pulled entirely out of the scope of the broker.
Then I have written a scheduled service (a normal spring bean which 
executes tasks with Executors ) which periodically checks the state
of the queues and eventually takes out messages from the store and sends
them to their ultimate destination.

James.Strachan wrote:
> So I think you should be able to do all of this with just a few
> message consumers / producers and no need to write a separate store or
> anything. I think the main use case for a separate message
> database/store is when you need to combine multiple mesasges together
> into a single aggregate (kinda like doing orchestration).

The problem I had was that I wasn't sure of how to introduce timing within a 
consumer, that is doing what a scheduled task does: polling the system's
state and reacting when this changes. 
A message listener is event driven by design, so it doesn't seem to fit 
the poll/react pattern, does it?

Do you think the above makes sense? I was a bit reluctant of using a
task aside of JMS, it seems to somehow break the model itself, but I
find a better way...

I would most appreciate a comment on this, thanks again.
View this message in context:
Sent from the ActiveMQ - User mailing list archive at

View raw message