activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken1000 <kendrum...@yahoo.com>
Subject Batching messages for better compression?
Date Wed, 14 Nov 2012 16:41:07 GMT
We've been using ActiveMq (as well as ServiceMix and Camel) for some time
now, and they all work great handling XML data (we generally use
non-persistent, non-transacted topics).  One area that does cause us
problems is the bandwidth when transmitting the data over a wire.  Sometimes
we're sending a lot of relatively small XML messages, with a handful of JMS
string properties describing the contents of each message.

We specify in our Camel configuration that the ActiveMQ broker should
compress the messages before sending, but they don't seem to compress down
very well because 1) we think the JMS properties aren't compressed, and 2)
they are relatively small messages without a lot of repetition in the text.

We've tried using the Camel "aggregator" pattern to batch message bodies
together before sending, which gives much better compression, but at the
cost of losing the properties attached to each message.  We could avoid
losing the properties by 1) aggregating messages based on common properties
(which results in smaller batches and less compression - an undesired
result), or 2) by aggregating each message's properties along with the
message bodies (which could then be used to reconstitute the individual JMS
messages with their properties after transmission), but it seems this would
require any consumer to have a complimentary de-aggregator in place -
another undesired result.

Our dream solution would be if the ActiveMQ broker could be configured to
batch and compress messages (including the properties) prior to
transmission, and then separate them back to individual messages to be read
by the subscriber's onMessage call, but I can't find this anywhere in the
documentation (prefetch doesn't seem to handle this, etc...).  

Is there something I've missed that would enable this?

Thanks!





--
View this message in context: http://activemq.2283324.n4.nabble.com/Batching-messages-for-better-compression-tp4659340.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message