camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiram Chirino" <hi...@hiramchirino.com>
Subject Re: Aggregator strategies (again)
Date Mon, 05 Nov 2007 15:30:09 GMT
On 11/5/07, James Strachan <james.strachan@gmail.com> wrote:
> On 05/11/2007, Hiram Chirino <hiram@hiramchirino.com> wrote:
> > On 11/5/07, James Strachan <james.strachan@gmail.com> wrote:
> > > BTW Alberto I've added your test case to the distro so folks can
> > > noodle on it and see if we can improve things some. See
> > > AlbertoAggregatorTest in the camel-core module.
> > >
> > > I commented out the use of the set batch size; as we don't really need
> > > to worry about that in this case, as we can just assume if the timeout
> > > fires after a few seconds we've got to the end of any possible batch.
> > >
> > > The tricky thing is knowing when you're at the end of the batch
> > > really. One simple solution would be to add some kinda predicate to
> > > detect batch-completion. For example  when we split messages we could
> > > record how many split messages there are and each messages' counter so
> > > that we know when we've aggregated them all together again?
> >
> > I wonder if the simplest way to implement the described split and
> > aggregate pattern could be to just combine the splitter and aggregator
> > into 1 delegate processor.  For example it would take it's input,
> > split it and forward each part to the delegate for processing.  Then
> > it would wait for the result of processing and aggregate those
> > results.  The delegate processor would know how many things it split
> > up so it would know how many things to aggregate.
>
> Thats a great idea for a nice simple solution that solves many
> problems :) Increasingly it might help if we can try to roll up
> smaller patterns into larger patterns (or protocols?) so combining a
> splitter and aggregator together into a single unit might help
> simplify things for folks (and make it easier to avoid making
> mistakes).
>
> I guess the downside of this approach is that only one large message
> can be processed at once. For example using Invoices and LineItems as
> an example; if you had a pool of processors, you could only let folks
> process one Invoice at once (as the thread would block before sending
> any other invoice's LineItems to be processed) but that might avoid
> complications.
>

Actually that would not be true.  It should be safe to call
Splitter/Aggregator concurrently since it's state would be local the
exchange being processed.  For example:

from("jms:queue:input").thread(5).splitAndAggregate(...)...

And the processing of the split parts would not have to happen in
sequentially either if the  Splitter/Aggregator was implemented an
AsyncProcessor.  It could do something like:

  ...splitAndAggregate(...).thread(5).processor(stepProcessor);

basically it would sequentially send all the steps to the thread
processor but those would return right away since an async thread
would be started to complete process each step.  The
Splitter/Aggregator  would then just listen to the async completion
events so that it could aggregate the results and return when all the
steps have been aggregated.



> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source SOA
> http://open.iona.com
>


-- 
Regards,
Hiram

Blog: http://hiramchirino.com

Mime
View raw message