activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From detla <jonathan.schm...@detla.de>
Subject Question concerning JMS/ActiveMQ Design
Date Tue, 05 Jul 2011 13:15:51 GMT
Hi all,

we currently consider the use of JMS implemented by ActiveMQ in the
following scenario:

A producer will produce roughly 300 to 400 rather large (20-30 MB) data
objects per minute (in total ~17000 objects). Seven different
processor-objects process the data objects in parallel. The setting is
purely local, so the embedded message-broker is sufficient. Durable messages
are not required. Naively you would assume, that the best setup will be one
non-durable topic, where the seven processors register as subscribers and
the producer sending the data objects to that topic. Everything within this
setting is purely java.

Our problem is the memory consumption of the setup. There is a limit of
roughly 100 of the data objects, which can be kept in memory at a given
time. Typically, the seven processors are roughly equally fast but there may
be situations, where one is lagging behind. In this case, the remaining fast
processors should proceed without decreasing speed, while for the lagging
processor, it is acceptable to complete its task with a delay.

Until now we have identified several options how to deal with this problem
and neither makes us really happy:
1) Intelligent data objects: The data objects get some intelligence and time
to live. Once the time to live is up, the objects starts persisting the
larger data chunks contained, thus freeing memory. Of course this solution
requires some rematerialization, once the object is needed again by the
lagging processor. This of course requires some tedious implementation (plus
the question as to how the data object gets notified, that it should persist
itself without interfering with gc).

2) (Ab)Use of JMS techniques: Try to achieve the following: Via priority
assure, that newest messages (and objects) are consumed first. If an old
object gets too old (or better: queue gets to long), the object is subject
to persisting via the JMS mechanisms. Question: is this a violation of the
design principles? Is it technically even possible?

3) Re-Aquiring data objects: Use the ActiveMQ mechanism to drop old
messages, via a second mechanism (simple java communication) re-request the
data-objects that were lost afterwards. This of course minimizes our chances
to ever make use of the distributed mechanisms of JMS and seems to involve a
lot of manual work.

4) Central observer: Use a central observer to notify about consumed
messages and filled queues. Controll persisting/dropping/redelivery of
messages via the central observer. Again: lots of manual work.


All in all our opinion is, that the requirements are somewhat standard and
we have the feeling, we might be missing some plain nice and easy way to
make use of standard-JMS-techniques (or ActiveMQ enhancements to the
standard). 

Has someone some idea where to start looking or some input to the
possibilities we already identified?

Kind regards,
Jonathan


--
View this message in context: http://activemq.2283324.n4.nabble.com/Question-concerning-JMS-ActiveMQ-Design-tp3645904p3645904.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message