camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taariq Levack <taar...@gmail.com>
Subject Re: Rate limit producer to match consumer
Date Wed, 31 Dec 2014 13:06:17 GMT
I just fired up a sample app based on the file-size CBR I mentioned, hope I
didn't misunderstand or oversimplify so take a look if you haven't solved
it yet.
https://github.com/levackt/samples/tree/master/file-size-cbr

Also take a look at the route throttling example which shows off the
policies nicely.
http://camel.apache.org/route-throttling-example.html

Taariq



On Tue, Dec 30, 2014 at 8:32 PM, Taariq Levack <taariql@gmail.com> wrote:

> How about a content based router that checks the file size and forwards
> large messages to one endpoint and smaller ones to another?
> Both are throttled or limited appropriately, and forward to your existing
> endpoint where the work is done.
>
> That last endpoint should have more consumers than either of the others.
> You can get more creative with the general idea, like dynamically setting
> the maximumRequestsPerPeriod[1] higher for the small file endpoint when
> there are no large files to process.
> [1] http://camel.apache.org/throttler.html
>
> Taariq
>
> > On 30 Dec 2014, at 13:41, Nathan Jones <nathan@ncjones.com> wrote:
> >
> > We are trying to improve the responsiveness of some bulk message
> processes such that a large batch does not flood a queue and prevent
> subsequent smaller batches getting through in a timely fashion. For
> example, a job to import millions of records from CSV may take an hour but
> a smaller job to import just one thousand records should be able to begin
> processing immediately in parallel instead of being sent to the back of a
> very long queue.
> >
> > This seems to me like the sort of thing Camel should be good at but I
> have so far not been about to see how this could be achieved. The idea we
> have in mind is to have a queue with limited size that will block when it
> is full so that the rate of queuing from a large batch would be limited to
> consumer capacity. Subsequent batches would have equal opportunity to get
> the next message on to the queue.
> >
> > At first I thought the Camel maxInflightExchanges property could be used
> for this but I don't think it has this affect. Is there a way a Camel route
> can inspect the size of the target queue to decide whether to suspend or
> resume?
> >
> > Perhaps a message broker can help solve this with either blocking queues
> or virtual aggregate queues but I haven't found these in RabbitMQ or
> ActiveMQ.
> >
> > Does anyone have any advice on a way solve this problem with Camel or
> otherwise?
> >
> > - Nathan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message