camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seth Call <>
Subject Re: Controlling how, and which messages, a JMS Consumer pulls
Date Mon, 04 Oct 2010 21:38:18 GMT
Hi Illtud,

Essentially I have no natural hierarchy.  The issue is that these
workers will generally interoping with 3rd-party services, meaning
it's very hard to know upfront what their runtime characteristics will
be.   We may find that one API of one 3rd-party fails alot and/or
takes a long time, and we decide we need to fire up many Workers that
are dedicated to that service so that we can improve concurrency and
prevent that one service from backlogging the whole system.

However, the whole point is that, after runnig this thing in
production, and realizing it makes sense to have jobs of type 'A, B,
C' lumped in usually with one worker type, and jobs D, E should always
have a dedicated worker, then you may then have a hierarchy like:


And then if I wanted a worker to do A, B, C, my path is clear.  If I
wanted to do A and D, then I have a problem, sure, but the idea here
is that I shouldn't need to once I know my 'realistic hierarchy' if
that makes sense.

Good suggestion, thanks very much.


On Mon, Oct 4, 2010 at 4:25 PM, Illtud Daniel <> wrote:
> On 04/10/10 21:46, Seth Call wrote:
>> It seems to me the right thing to do is to have one queue per
>> unit-of-work type.  So if you are a worker and support 2 types of
>> work, then you'd ultimately be listening to two queues. But, once you
>> started working on one task pulled from one of the queues, it seems
>> you'd want to dynamically stop listening to the other queue--and
>> conversely, once the task is done, you'd start listening on that queue
>> again.  But right away I'm concerned how one could guarantee just one
>> message is read from either queue at a time.
> I think this could be done by using wildcards. If you have jobs
> of type A, B, C, D & E, and jobs A, B and C are processorder
> jobs (for example) and D & E are dispatchorder jobs, and you
> have 3 consumers, X, Y & Z. X can do processorders and dispatchorder
> -type jobs, Y can do processorders, and Z can do dispatchorder,
> then you can have queues:
> queue.processorder.A
> queue.processorder.B
> queue.processorder.C
> queue.dispatchorder.D
> queue.dispatchorder.E
> Then X could subscribe to queue.*, Y to queue.processorder.*
> and Z to queue.dispatchorder.*
> If your workers are single-threaded, and do their work in an onmessage
> (Java) method, then they'll automatically do the stuff you want
> (stop listening, not block other consumers) while they work on the
> message.
>> Another scenario is if no workers are currently available that can
>> handle a particular type of task because they are all busy, but there
>> are other things to do in the queue.
> My solution would work for that, since there are separate queues.
> The only issue is if your task hierarchy doesn't break down handily
> like mine does (eg if Z could do A & E, but not the other tasks), but
> you'd have pretty odd workers to have that scenario (or I don't have
> much imagination).
> --
> Illtud Daniel                       
> Prif Swyddog Technegol                          Chief Technical Officer
> Llyfrgell Genedlaethol Cymru                  National Library of Wales

View raw message