camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Krasser" <...@martin-krasser.de>
Subject AW: Reliable resequencing
Date Sat, 06 Sep 2008 08:15:30 GMT
Hi Dirk,

the short answer is: no, the latest patch contained CAMEL-126 doesn't solve
the problem that you might loose messages if there a system failure. The
long answer is: it is only a first step into that direction:  

a) The patch ensures that reading a message from an endpoint and delivery of
pending/reordered messages occurs within a single thread (in Camel 1.4 there
is still a separate thread doing the delivery). However, the transaction
from the source endpoint currently isn't propagated yet.
b) The patch changes the current implementation of the stream resequencer to
work with a polling consumer in the same way as the batch resequencer does
(btw, the batch resequencer currently isn't suitable for reliable
resequencing as well, from what I can see from the source code). 

To make both resequencers reliable the following extensions are required:

- Batch resequencer: read n messages within a single transaction (instead of
n transactions as done currently in 1.4), reorder them, send them to a
destination and commit the transaction. This can easily done within a single
thread.
- Stream resequencer: the situation is more tricky here because there is a
continuous stream of messages without having candidates for transaction
boundaries (in contrast to the batch resequencer where the batch size can be
used to determine a transaction boundary). I'm still not 100% sure how to
solve this problem but a possible solution might roughly look like this:
Read messages from a source endpoint with n threads, suspend them until they
have been re-ordered based on the message they have read and resume/execute
them sequentially after their message becomes eligible for delivery. After
delivery of the message each thread commits the transaction it started.
Consequently, each message is delivered to a destination within a separate
transaction (in contrast to the batch resequencer where several messages can
be delivered within a single transaction) but the resequencing is reliable
too.

So far I had no time to work on these extensions. The initial patch of
CAMEL-126 is planned to be applied with Camel 1.5, so I'll try to get the
extensions done in the 1.6-SNAPSHOT because I also see the need for reliable
stream resequencing in one of my upcoming projects.

Even if it's not an immediate help for your problem, do the proposed
extensions make sense to you? Can you see any alternatives how to solve the
reliable (stream) resequencing problem? 

Cheers,
Martin

> -----Urspr├╝ngliche Nachricht-----
> Von: dirk.mahler [mailto:dirk.mahler@buschmais.com]
> Gesendet: Freitag, 5. September 2008 15:16
> An: camel-user@activemq.apache.org
> Betreff: Reliable resequencing
> 
> 
> Hi,
> 
> we integrated Camel 1.4 as JBI component into ServiceMix and tried to use
> the stream resequencer to reconstruct the "natural" order of our messages.
> 
> The routing on ServiceMix (SMX) level is defined as follows:
> 
> SMX transactional JMS/JCA consumer -> Camel JBI Component -> SMX Bean SE
> 
> The Camel JBI Component is configured to provide resequencing the
> following
> way (I left out some message body conversion stuff):
> 
> from("jbi:endpoint:...")
>   .resequencer(sequenceExpression)
>   .stream(config)
>   .to("jbi:endpoint:.../beanEndpoint");
> 
> Resequencing seems to work but according to some simple tests it cannot be
> guaranteed that all messages will reach the final consumer (in our case
> the
> bean endpoint) if there's a system failure: Camel acknowlegdes the message
> exchange to the JMS Consumer but then holds the messages in a non-
> persistent
> queue before delivering them to the SMX bean endpoint. We found an
> existing
> JIRA issue (CAMEL-126, https://issues.apache.org/activemq/browse/CAMEL-
> 126)
> which at a first glance seems to address our concerns by thinking about
> synchronous dispatching - would this solve our problem or is there already
> another solution?
> 
> 
> 
> -----
> Dirk Mahler
> Senior Consultant
> 
> -----------------------------------------------------------------
> buschmais GbR
> Inhaber  Torsten Busch, Frank Schwarz, Dirk Mahler, Tobias Israel
> Adresse  buschmais GbR, Leipziger Stra├če 93, 01127 Dresden
> Telefon  +49 (0) 351 3209 23-0
> Fax       +49 (0) 351 3209 23-29
> Telefon  +49 (0) 1577 198 295 0
> E-Mail   dirk.mahler@buschmais.com
> Internet http://www.buschmais.de
> -----------------------------------------------------------------
> --
> View this message in context: http://www.nabble.com/Reliable-resequencing-
> tp19331003s22882p19331003.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



Mime
View raw message