camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerry Williamson (JIRA)" <>
Subject [jira] [Updated] (CAMEL-7521) Provide an option for unsynchronized aggregation when splitter is streaming and not parallel
Date Wed, 18 Jun 2014 22:12:25 GMT


Jerry Williamson updated CAMEL-7521:


attached a patch that provides an unsynchronizedSequentialAggregation option on MulticastProcessor
and exposes it through the SplitterDefinition.
I tried backwardly compatible and provide opt-in to the behavior for the option.

I've also attached a "test".
It's not really a proper test that asserts conditions but it will print out the elapsed time
of running concurrent in-flight requests with synchronized and unsynchronized aggregation.
It will demonstrate the increasing impact of the synchronization as the number of concurrent
in-flight requests grows.

> Provide an option for unsynchronized aggregation when splitter is streaming and not parallel
> --------------------------------------------------------------------------------------------
>                 Key: CAMEL-7521
>                 URL:
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-core
>    Affects Versions: 2.13.2
>            Reporter: Jerry Williamson
>         Attachments:, camel.multicast.processor.patch
> We use a splitter route that splits very large files.
> The splits need to be aggregated in order.
> The splitter is therefore configured with streaming=true and parallelProcessing=false.
> When there are multiple in-flight exchanges on the route (i.e. different files are being
processed) the end-to-end processing time for each file is significantly impacted because
each aggregation call is synchronized in MulticastProcessor.doAggregate method..
> It is not uncommon, in our environment, to have thousands of splits per file and the
synchronization is significantly impacting the throughput of the route.
> I propose that an option be added (to splitter at least) to allow for an unsynchronized
aggregation. The option would default to false for backward compatibility and would possibly
be ignored if parallelProcessing = true.
> The body of the MulticastProcess.doAggregate method could be moved to a new, unsynchronized
method (say doAggregateInternal). The MulticastProcessor.doAggregate method could remain synchronized
(again for backward compatibility) but the body of that method would then call the new unsynchronized
doAggregateInternal method. In this way, existing code would remain synchronized.
> The splitter, then, could call the unsynchronized doAggregateInternal method when the
unsynchronizedAggregation option was true.

This message was sent by Atlassian JIRA

View raw message