camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quinn Stevenson <qu...@pronoia-solutions.com>
Subject Re: Best way to build a wait/retry loop with Camel?
Date Thu, 03 Nov 2016 16:22:25 GMT
I’ve done something similar to option 3 before, the difference being that I shutdown the
route using the Control Bus after I’d processed each message.

The other option I can think of is use two routes and two queues.  The first route would try
and process the “waiting” messages, and if the message cannot be successfully processed,
publish it to another queue.  The second route would simply read the message from the second
queue and put them back on the “waiting” queue.  This second route could be controlled
with the CronScheduledRoutePolicy so it would only run once a day.

If you’re using ActiveMQ, you could try using the scheduler (http://activemq.apache.org/delay-and-schedule-message-delivery.html
<http://activemq.apache.org/delay-and-schedule-message-delivery.html>) to delay the
delivery - so you’re route could always be running.  However, instead of rolling-back the
message, you’d publish it back to the same queue and let ActiveMQ schedule when it would
be re-delivered to you.

HTH


> On Nov 3, 2016, at 9:46 AM, Stephan Burkard <sburkard@gmail.com> wrote:
> 
> Hi Camel users
> 
> TLDR:
> - how to implement wait/retry loop?
> - messages "wait" in a queue
> - each message should be reprocessed only once per interval (let's say per
> day)
> - after reprocessing a message either returns to the waiting queue or
> continues (goes out of loop)
> - after [maxRetry] attempts it is moved away (goes out of loop)
> 
> 
> Full story:
> 
> As part of a typical processing workflow, I have a queue with "waiting"
> messages. These messages are waiting for data of another system and I have
> no idea when the data arrives. Therefore I want to reprocess the messages
> in an interval, let's say once a day. If the data is still missing, they go
> again to the waiting queue.
> To move messages away that would wait forever, I set and check a maxRetry.
> 
> How would I best implement this? Some thoughts I made:
> 
> 1. Just throw an exception if the data is missing. The message is
> redelivered.
>   Problem: The message is instantly redelivered. So it does the retries
> within milliseconds. I cannot customize the interval.
> 
> 2. Use the Camel delayer.
>   Works good in general (my current solution). I can even calculate the
> delay based on the JMSTimestamp to take into account the time a message
> waits to be consumed.
>   Problem: For an interval of 1 day, a message is permanently in
> processing state. The consumer is blocked. When I shutdown the container,
> Camel waits 5 minutes and after restarting the message goes to the DLQ.
> 
> 3. Use a CronScheduledRoutePolicy to start/stop the route in the given
> interval.
>   Problem: When I start the route once a day for 5 minutes, the messages
> with missing data are circulating until the route is stopped again. So they
> try to find the data multiple times per second and instantly reach the
> maxRetry. They must try only once per interval.
> 
> 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp <
> (currentTime - intervalTime)"
>   Problem: The from-Endpoint URI seems to be a constant, I can't make the
> selector dynamic.
> 
> 5. Use the Camel Throttler
>   Problem: The throttler sets a bandwith, that is not what we need.
> 
> 6. Use a Timer from-Endpoint and hand over to a bean that consumes the
> messages (newest idea, not yet tried).
>   I guess I could manage to consume all waiting messages just once (even
> if they already return before I am finished).
>   Problems I can think of: It is not very nice to build another JMS
> consumer (beside Camel), but I could use Spring JMS Template.
> 
> 
> Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right
> way) for this case. Number 2 is my current solution, number 6 a new idea.
> Any recommendations or other suggestions how to build this wait/retry loop?
> 
> Thanks a lot
> Stephan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message