activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Consuming a group of messages in a single transaction
Date Fri, 28 Jul 2006 07:58:03 GMT
On 7/28/06, nlif <naaman@dbnet.co.il> wrote:
>
> Thanks, James.
>
> Sending just the URI is not an option in my case, since the participating
> applications do not share a file server (or a database, for that matter).
>
> As for the second suggestion, just to make sure I understand: do you mean
> that I should implement the equivalent of the ActiveMQ streams feature, by
> splliting the files into JMS ByteMessages and then include them in the
> MessageGroup and in the transaction? (btw I've looked at the Streams code,
> and I know this is pretty much what it does, only that the input stream
> auto-acknowledges each chunk, right?).

I mean split your transaction from sending many Mbs of data in 1
transaction to splitting the file into more manageable chunks and
having lots of littler transactions.

The other option is just to buy a big box with heaps of RAM :)


> Now, could there be a different approach to all of this? My whole problem
> stems from the fact I don't want to acknowledge ANY of the items (messages
> and/or files) until I have ALL of them. The reason is simple: if my cosumer
> crashes half-way, some data is lost. However, trying to sync the acks for
> all different items, that may be passed via different mechanisms, makes
> things very complicated for me.
>
> Is there another way? Is it possible to go back to simple MessageListeners
> and InputStream consumers, which acknowledge every item separately, but have
> a different mechanism for ensuring their  state is recoverable? Maybe I
> should go for a consumer that can persist its data? Or is this even harder
> to achieve?

Why can't you acknowledge in batches; say a few 100K at a time?


> This seems to me like something many applications might require. Am I really
> the only one faced with a scenario in which a transmission is composed of
> several items, that cannot be sent as a single JMS message? Is there no
> generic solution for this?

For most of the customers I work with, their transaction boundaries
are reasonably small with 10-100K of data per transaction. If you need
to process something larger you can often split it up into smaller
chunks.
-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message