camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <>
Subject Re: Split an exchange, process pieces in parallel and then wait for all to complete
Date Wed, 09 Jun 2010 04:15:46 GMT

The Camel in Action book chapters 8 and 10 covers the Splitter EIP in
detail and how to use it with concurrency as well to split in

But there is also some bits of details at the Camel wiki page

The Splitter has a build in aggregator so it can be used to create the
single outgoing file you want.

On Wed, Jun 9, 2010 at 1:12 AM, jonathanq <> wrote:
> Hello,
> I am trying to get my head around how to construct a specific route in camel
> 2.3.  I am creating a process that listens on a "trigger" directory for a
> new file which starts the process.
> Once it receives the "go" message.  It will then look at a directory and
> start processing all of the files in that directory.  It is going to split
> those files into smaller files and compress them.
> Once it is finished processing all of the files - we need to send a single
> output message saying its done.
> Here is the general idea:
> from("file://trigger").process(getListOfFilesInDirectoryProcessor()).split(body()).threads(10).process(getFileSplitProcessor()).log("processing
> of all input files complete")
> Obviously in the above example - we get a log message per input file, which
> we don't want.
> Another idea was to use SEDA queues and an aggregator.  This works - but it
> relies on knowing how many input files there were for the batchSize() in the
> aggregator.  It also introduces the batchtimeout as well.  It is not ideal -
> so I am wondering if there is a better way.
> from("file://trigger")
> .process(getListFilesInDirectoryProcessor())
> .split(body())
> .to("seda:atomicFiles");
> from("seda:atomicFiles")
> .threads(10)
> .process(getFileSplitProcessor())
> .log("File thread completed")
> .to("seda:aggregate");
> from("seda:aggregate")
> .aggregate(header("extractId"))
>  .batchSize(2)  // this needs to be the same as the number of input files
>  .batchTimeout(10000000)
>  .log("Processing of input files complete.");
> I have tried to get my head around the new async package in Camel 2.x - but
> I am having trouble understanding how to get it to do a request/reply.
> Is there an easier way to do what I want?  The Async documentation goes into
> detail on how to use callbacks - none of which seem to be in DSL routes.
> The threads() function seems to be the DSL version...but it doesn't describe
> how to do the request/reply.  The example route just splits to multiple
> threads and its done.
> What I am doing seems to be a common problem - get 1 input message...split
> the work into smaller chunks to be processed simultaneously and then when
> all pieces are done - continue on.  So this EIP:
> The difference here is that I want the same processing to occur..just
> multi-threaded. Whereas that EIP is more for different processing on each
> piece.
> Thanks!
> Jonathan
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

Claus Ibsen
Apache Camel Committer

Author of Camel in Action:
Open Source Integration:

View raw message