camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Jones <nat...@ncjones.com>
Subject Rate limit producer to match consumer
Date Tue, 30 Dec 2014 11:41:05 GMT
We are trying to improve the responsiveness of some bulk message 
processes such that a large batch does not flood a queue and prevent 
subsequent smaller batches getting through in a timely fashion. For 
example, a job to import millions of records from CSV may take an hour 
but a smaller job to import just one thousand records should be able to 
begin processing immediately in parallel instead of being sent to the 
back of a very long queue.

This seems to me like the sort of thing Camel should be good at but I 
have so far not been about to see how this could be achieved. The idea 
we have in mind is to have a queue with limited size that will block 
when it is full so that the rate of queuing from a large batch would be 
limited to consumer capacity. Subsequent batches would have equal 
opportunity to get the next message on to the queue.

At first I thought the Camel maxInflightExchanges property could be used 
for this but I don't think it has this affect. Is there a way a Camel 
route can inspect the size of the target queue to decide whether to 
suspend or resume?

Perhaps a message broker can help solve this with either blocking queues 
or virtual aggregate queues but I haven't found these in RabbitMQ or 
ActiveMQ.

Does anyone have any advice on a way solve this problem with Camel or 
otherwise?

  - Nathan

Mime
View raw message