camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: Aggregator strategies (again)
Date Mon, 05 Nov 2007 15:38:35 GMT
Oooh! I forgot about that! Thats a great idea! :) Lets do it!

I guess a split/aggregate pair is gonna be very common.

BTW you could argue that split/aggregate is kinda like map/reduce too...


On 05/11/2007, Hiram Chirino <hiram@hiramchirino.com> wrote:
> On 11/5/07, James Strachan <james.strachan@gmail.com> wrote:
> > On 05/11/2007, Hiram Chirino <hiram@hiramchirino.com> wrote:
> > > On 11/5/07, James Strachan <james.strachan@gmail.com> wrote:
> > > > BTW Alberto I've added your test case to the distro so folks can
> > > > noodle on it and see if we can improve things some. See
> > > > AlbertoAggregatorTest in the camel-core module.
> > > >
> > > > I commented out the use of the set batch size; as we don't really need
> > > > to worry about that in this case, as we can just assume if the timeout
> > > > fires after a few seconds we've got to the end of any possible batch.
> > > >
> > > > The tricky thing is knowing when you're at the end of the batch
> > > > really. One simple solution would be to add some kinda predicate to
> > > > detect batch-completion. For example  when we split messages we could
> > > > record how many split messages there are and each messages' counter so
> > > > that we know when we've aggregated them all together again?
> > >
> > > I wonder if the simplest way to implement the described split and
> > > aggregate pattern could be to just combine the splitter and aggregator
> > > into 1 delegate processor.  For example it would take it's input,
> > > split it and forward each part to the delegate for processing.  Then
> > > it would wait for the result of processing and aggregate those
> > > results.  The delegate processor would know how many things it split
> > > up so it would know how many things to aggregate.
> >
> > Thats a great idea for a nice simple solution that solves many
> > problems :) Increasingly it might help if we can try to roll up
> > smaller patterns into larger patterns (or protocols?) so combining a
> > splitter and aggregator together into a single unit might help
> > simplify things for folks (and make it easier to avoid making
> > mistakes).
> >
> > I guess the downside of this approach is that only one large message
> > can be processed at once. For example using Invoices and LineItems as
> > an example; if you had a pool of processors, you could only let folks
> > process one Invoice at once (as the thread would block before sending
> > any other invoice's LineItems to be processed) but that might avoid
> > complications.
> >
>
> Actually that would not be true.  It should be safe to call
> Splitter/Aggregator concurrently since it's state would be local the
> exchange being processed.  For example:
>
> from("jms:queue:input").thread(5).splitAndAggregate(...)...
>
> And the processing of the split parts would not have to happen in
> sequentially either if the  Splitter/Aggregator was implemented an
> AsyncProcessor.  It could do something like:
>
>   ...splitAndAggregate(...).thread(5).processor(stepProcessor);
>
> basically it would sequentially send all the steps to the thread
> processor but those would return right away since an async thread
> would be started to complete process each step.  The
> Splitter/Aggregator  would then just listen to the async completion
> events so that it could aggregate the results and return when all the
> steps have been aggregated.
>
>
>
> > --
>
> > James
> > -------
> > http://macstrac.blogspot.com/
> >
> > Open Source SOA
> > http://open.iona.com
> >
>
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>


-- 
James
-------
http://macstrac.blogspot.com/

Open Source SOA
http://open.iona.com

Mime
View raw message