camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bilgin Ibryam <bibr...@gmail.com>
Subject Re: Best way to build a wait/retry loop with Camel?
Date Sat, 12 Nov 2016 19:08:48 GMT
Hi Stephan,

I've only read your short version of the question.
I've implemented such a delayed/persisted retry mechanism using
ActiveMQ and described in Camel Design Patterns book, Retry Pattern
chapter.

HTH,

On 3 November 2016 at 15:46, Stephan Burkard <sburkard@gmail.com> wrote:
> Hi Camel users
>
> TLDR:
> - how to implement wait/retry loop?
> - messages "wait" in a queue
> - each message should be reprocessed only once per interval (let's say per
> day)
> - after reprocessing a message either returns to the waiting queue or
> continues (goes out of loop)
> - after [maxRetry] attempts it is moved away (goes out of loop)
>
>
> Full story:
>
> As part of a typical processing workflow, I have a queue with "waiting"
> messages. These messages are waiting for data of another system and I have
> no idea when the data arrives. Therefore I want to reprocess the messages
> in an interval, let's say once a day. If the data is still missing, they go
> again to the waiting queue.
> To move messages away that would wait forever, I set and check a maxRetry.
>
> How would I best implement this? Some thoughts I made:
>
> 1. Just throw an exception if the data is missing. The message is
> redelivered.
>    Problem: The message is instantly redelivered. So it does the retries
> within milliseconds. I cannot customize the interval.
>
> 2. Use the Camel delayer.
>    Works good in general (my current solution). I can even calculate the
> delay based on the JMSTimestamp to take into account the time a message
> waits to be consumed.
>    Problem: For an interval of 1 day, a message is permanently in
> processing state. The consumer is blocked. When I shutdown the container,
> Camel waits 5 minutes and after restarting the message goes to the DLQ.
>
> 3. Use a CronScheduledRoutePolicy to start/stop the route in the given
> interval.
>    Problem: When I start the route once a day for 5 minutes, the messages
> with missing data are circulating until the route is stopped again. So they
> try to find the data multiple times per second and instantly reach the
> maxRetry. They must try only once per interval.
>
> 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp <
> (currentTime - intervalTime)"
>    Problem: The from-Endpoint URI seems to be a constant, I can't make the
> selector dynamic.
>
> 5. Use the Camel Throttler
>    Problem: The throttler sets a bandwith, that is not what we need.
>
> 6. Use a Timer from-Endpoint and hand over to a bean that consumes the
> messages (newest idea, not yet tried).
>    I guess I could manage to consume all waiting messages just once (even
> if they already return before I am finished).
>    Problems I can think of: It is not very nice to build another JMS
> consumer (beside Camel), but I could use Spring JMS Template.
>
>
> Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right
> way) for this case. Number 2 is my current solution, number 6 a new idea.
> Any recommendations or other suggestions how to build this wait/retry loop?
>
> Thanks a lot
> Stephan



-- 
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam

Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475

Mime
View raw message