activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Davies <rajdav...@gmail.com>
Subject Re: Batching messages for better compression?
Date Thu, 15 Nov 2012 12:25:16 GMT
There's been some discussion on batches for JMS2 - see
http://java.net/jira/browse/JMS_SPEC-36

On 14 November 2012 21:13, Gary Tully <gary.tully@gmail.com> wrote:

> Auto batching does not exist at the moment but it would be a nice
> enhancement.
> ie: configure some sort of client side pause time or size limit so
> that the transport would wait for a batch and send all at once in one
> compressed block.
> It would require an update to openwire to indicate the start/end of a
> compressed batch or a new batch command, but it is doable.
> Please open a jira ticket to capture this. It definatly warrents some
> investigation.
> Either it is something that activemq can do under the hood or
> something we can implement via plugins and filters.
>
> On 14 November 2012 16:41, Ken1000 <kendrum100@yahoo.com> wrote:
> > We've been using ActiveMq (as well as ServiceMix and Camel) for some time
> > now, and they all work great handling XML data (we generally use
> > non-persistent, non-transacted topics).  One area that does cause us
> > problems is the bandwidth when transmitting the data over a wire.
>  Sometimes
> > we're sending a lot of relatively small XML messages, with a handful of
> JMS
> > string properties describing the contents of each message.
> >
> > We specify in our Camel configuration that the ActiveMQ broker should
> > compress the messages before sending, but they don't seem to compress
> down
> > very well because 1) we think the JMS properties aren't compressed, and
> 2)
> > they are relatively small messages without a lot of repetition in the
> text.
> >
> > We've tried using the Camel "aggregator" pattern to batch message bodies
> > together before sending, which gives much better compression, but at the
> > cost of losing the properties attached to each message.  We could avoid
> > losing the properties by 1) aggregating messages based on common
> properties
> > (which results in smaller batches and less compression - an undesired
> > result), or 2) by aggregating each message's properties along with the
> > message bodies (which could then be used to reconstitute the individual
> JMS
> > messages with their properties after transmission), but it seems this
> would
> > require any consumer to have a complimentary de-aggregator in place -
> > another undesired result.
> >
> > Our dream solution would be if the ActiveMQ broker could be configured to
> > batch and compress messages (including the properties) prior to
> > transmission, and then separate them back to individual messages to be
> read
> > by the subscriber's onMessage call, but I can't find this anywhere in the
> > documentation (prefetch doesn't seem to handle this, etc...).
> >
> > Is there something I've missed that would enable this?
> >
> > Thanks!
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/Batching-messages-for-better-compression-tp4659340.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message