camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quinn Stevenson <qu...@pronoia-solutions.com>
Subject Re: Best way to build a wait/retry loop with Camel?
Date Fri, 04 Nov 2016 16:48:13 GMT
James brings up a very good point.  

I know ActiveMQ isn’t designed for long-term messages, and it’s bit me a couple of times
with KahaDB.  What happened was I had quite a few old messages, but they were spaced out over
time.  KahaDB doesn’t compact files, and it can’t delete files until all the messages
in the files are processed.  Long story short - we ran out of disk.

We addressed this part by using mKahaDB - it didn’t really “fix” the problem, but the
nature of our messaging is such that we could live with it.

> On Nov 4, 2016, at 10:34 AM, James Green <james.mk.green@gmail.com> wrote:
> 
> Are you unable to move the messages into a database for "long-term"
> storage? That's what we do, and we simply scan for messages due to be
> "re-sent" and re-submit them back into the queue.
> 
> I do not believe ActiveMQ was really intended to hold many messages for
> long term storage, hence we use something that was.
> 
> 
> On 3 November 2016 at 15:46, Stephan Burkard <sburkard@gmail.com> wrote:
> 
>> Hi Camel users
>> 
>> TLDR:
>> - how to implement wait/retry loop?
>> - messages "wait" in a queue
>> - each message should be reprocessed only once per interval (let's say per
>> day)
>> - after reprocessing a message either returns to the waiting queue or
>> continues (goes out of loop)
>> - after [maxRetry] attempts it is moved away (goes out of loop)
>> 
>> 
>> Full story:
>> 
>> As part of a typical processing workflow, I have a queue with "waiting"
>> messages. These messages are waiting for data of another system and I have
>> no idea when the data arrives. Therefore I want to reprocess the messages
>> in an interval, let's say once a day. If the data is still missing, they go
>> again to the waiting queue.
>> To move messages away that would wait forever, I set and check a maxRetry.
>> 
>> How would I best implement this? Some thoughts I made:
>> 
>> 1. Just throw an exception if the data is missing. The message is
>> redelivered.
>>   Problem: The message is instantly redelivered. So it does the retries
>> within milliseconds. I cannot customize the interval.
>> 
>> 2. Use the Camel delayer.
>>   Works good in general (my current solution). I can even calculate the
>> delay based on the JMSTimestamp to take into account the time a message
>> waits to be consumed.
>>   Problem: For an interval of 1 day, a message is permanently in
>> processing state. The consumer is blocked. When I shutdown the container,
>> Camel waits 5 minutes and after restarting the message goes to the DLQ.
>> 
>> 3. Use a CronScheduledRoutePolicy to start/stop the route in the given
>> interval.
>>   Problem: When I start the route once a day for 5 minutes, the messages
>> with missing data are circulating until the route is stopped again. So they
>> try to find the data multiple times per second and instantly reach the
>> maxRetry. They must try only once per interval.
>> 
>> 4. Use a JMS selector as from-Endpoint that selects "JMSTimestamp <
>> (currentTime - intervalTime)"
>>   Problem: The from-Endpoint URI seems to be a constant, I can't make the
>> selector dynamic.
>> 
>> 5. Use the Camel Throttler
>>   Problem: The throttler sets a bandwith, that is not what we need.
>> 
>> 6. Use a Timer from-Endpoint and hand over to a bean that consumes the
>> messages (newest idea, not yet tried).
>>   I guess I could manage to consume all waiting messages just once (even
>> if they already return before I am finished).
>>   Problems I can think of: It is not very nice to build another JMS
>> consumer (beside Camel), but I could use Spring JMS Template.
>> 
>> 
>> Numbers 1, 3, 4 and 5 simply do not work (or I don't use them the right
>> way) for this case. Number 2 is my current solution, number 6 a new idea.
>> Any recommendations or other suggestions how to build this wait/retry loop?
>> 
>> Thanks a lot
>> Stephan
>> 


Mime
View raw message