camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: splitter/aggregator scenatio
Date Mon, 05 Nov 2007 15:29:49 GMT
On 31/10/2007, Roman Kalukiewicz <roman.kalukiewicz@gmail.com> wrote:
> Hello!
>
> I wanted to implement splitter/aggregator scenario and I face following problem:
>
> When I split the original message I would like to aggregate results in
> the same order - to do this I have to somehow know which message was
> first, second, .. then I can use resequencer at the end to order them
> and aggregate in right order.
> Now I don't see any way to 'number' those messages that go from splitter.

See below... I've patched the Splitter...


> Another thing is aggregation - I' affraid I cannot use aggregator
> because few features of aggregator.
> In such scenario I cannot dynamically specify how many messages to
> aggregate (If I splitted to n parts i want to merge n responses) so
> I'll have to wait until timeout.

Yeah - we should allow some kinda Predicate to determine whether a
batch has been completed or not.

Right now the implementation is kinda simplistic; we could change it
to send a fully aggregated message as soon as a set of messages have
been fully aggregated - and not really have the concept of a batch.
Though the worry is we end up with a monster RAM collection of
messages waiting to be aggregated.

As mentioned in the other thread, we may want to use a database to
deal with massive numbers of messages which aggregate over a long
period of time.
http://www.nabble.com/Re%3A-Aggregator-strategies-%28again%29-p13587184s22882.html


> Moreover timeout in BatchProcessor is something that is not started by
> first message from the bath - timeouts just tick all the time and they
> don't care when messages arrive.
>
> Is there any ready solution for my problem or it just requires
> creating new classes that are similar to BatchProcessor where timeout
> is 'event based' and we can use some Expression to define number of
> elements to collect.

Yeah - I think we need to refactor BatchProcessor to use some kinda
pluggable predicate to determine when a batch is completed.

I've just added a little isBatchCompleted(index) method so a derived
class could add some extra predicates to decide if a batch is
completed.

I've also had a little experiment adding a Predicate to the Aggregator
and making a PredicateAggregatorCollection which only adds exchanges
to the collection to be dispatched when the predicate matches. I've no
idea if it works or anything - I've not written any tests yet :)

I wondered if it'd be of any use to you as a base to build on to
experiment with aggregations using custom predicates to determine the
completion? If its of no use, its easy to delete later :)

I've also raised this JIRA to track the predicate based Aggregator idea
http://issues.apache.org/activemq/browse/CAMEL-207


> Moreover maybe it is possible to set some standard header in splitter
> to mark the number of the message produced? In my case to know how
> many messages we have can be evaluated using xpath and set as a header
> before splitter.

Thats a great idea!

I've raised
http://issues.apache.org/activemq/browse/CAMEL-206

and just committed a test case and patch for this one.


> Thank you for all your thoughts,

And thank you! :)
-- 
James
-------
http://macstrac.blogspot.com/

Open Source SOA
http://open.iona.com

Mime
View raw message