activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Duplicating Large Messages
Date Mon, 03 Jul 2006 11:15:48 GMT
On 7/2/06, nlif <naaman@dbnet.co.il> wrote:
> Hello all,
>
> We need to be able to handle very large messages (200 Mb and more), which
> should be sent to multiple consumers (several hundreds of consumers). We
> would like to know what is the memory foot-print in the broker, when the
> same message is placed in several queues. Is the message duplicated in
> memory?

Typically yes. Also on consumers they typically buffer up messages as
well so you might wanna reduce prefetch sizes to something really
small (say 1)...

http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html


> Let me elaborate on the reasons for this question:
>
> At first, we have considered using a Topic, so that there is only a single
> copy of each message, regardless of how many consumers should get it.
> However, we need to be able to deliver to consumers who are not connected at
> the time of sending. We know we can use durable subscription, but we assume
> this results in a larger memory foot-print, as the broker keeps messages for
> some time.

The broker keeps messages on disk for a longer time but generally
doesn't have to keep them all in RAM


> Another problem with the Topic approach, is that this requires we include a
> recipient-list in the message (as JMS message properties), and provide the
> consumers with a JMS selector, so that consumers will only consume messages
> that are sent to them.

Don't you just send to the topic and let consumers decide what topics
to consume? Or is it that there is no easy way to introduce a logical
topic so want to put the consumer IDs on the message.


> However, this is a potential security problem, and we
> prefer using a queue per consumer, and a recipient-list based router, which
> will duplicate the original message and place a copy of it for each
> consumer, in that consumer's queue.

OK; its a fair bit slower to use many messages in different queues
than one message in a topic (as durable topics are specifically
optimised to do one-to-many delivery) but it should work. Though for
200M messages you'll generally need a monster heap.


> The above led us to the conculusion we need a Queue per consumer, and to the
> question - does ActiveMQ duplicate a message when placing it in several
> Queues, or is it internally a single copy?

Its a copy. For durable topics its shared.


> Another question - is the ActiveMQ feature of large input and output streams
> relevant for our problem?

Yes, definitely


> We would appreciate any input - this is a major show-stopper for us.

Given the requirements I'd recommend using the input/output streams
with multiple queues and using durable queues.
http://incubator.apache.org/activemq/jms-streams.html

which allows massive 'messages' to be sent to consumers without
requiring monster heaps. Since things are persistent we can evict
things from RAM to avoid having huge heaps.

If you are worried about performance & don't need the atomic fail-fast
guarrentee of persistent messaging you could enable async sends
http://incubator.apache.org/activemq/async-sends.html

-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message