Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (nike.apache.org: domain of claus.ibsen@gmail.com
 designates 74.125.82.173 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZIlGUwyMQXxd8iyDWxNacboWKyFyD22YNVx79kfNhYvrfFQOCLkbazzRJTUsX+emV1
         jbfXK0ocyWe4W7f8qLL/QU+QzXU0lGGSNOVg0gWjugVsGftwRBQ3oTnbPtfmPSUmirZU
         odpGaS3aE0UvyctTl1mFZdeF+97rniV0AAPT8=
MIME-Version: 1.0
In-Reply-To: <28824028.post@talk.nabble.com>
References: <28824028.post@talk.nabble.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Wed, 9 Jun 2010 06:15:46 +0200
Message-ID: <AANLkTinUfMUgO1pzX0Dv8yahW40ayRNQh8lj1rzqK3P9@mail.gmail.com>
Subject: Re: Split an exchange, process pieces in parallel and then wait for
	all to complete
To: users@camel.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi

The Camel in Action book chapters 8 and 10 covers the Splitter EIP in
detail and how to use it with concurrency as well to split in
paralllel.

But there is also some bits of details at the Camel wiki page
http://camel.apache.org/splitter.html

The Splitter has a build in aggregator so it can be used to create the
single outgoing file you want.


On Wed, Jun 9, 2010 at 1:12 AM, jonathanq <jquail@abebooks.com> wrote:
>
> Hello,
>
> I am trying to get my head around how to construct a specific route in ca=
mel
> 2.3. =A0I am creating a process that listens on a "trigger" directory for=
 a
> new file which starts the process.
>
> Once it receives the "go" message. =A0It will then look at a directory an=
d
> start processing all of the files in that directory. =A0It is going to sp=
lit
> those files into smaller files and compress them.
>
> Once it is finished processing all of the files - we need to send a singl=
e
> output message saying its done.
>
> Here is the general idea:
>
> from("file://trigger").process(getListOfFilesInDirectoryProcessor()).spli=
t(body()).threads(10).process(getFileSplitProcessor()).log("processing
> of all input files complete")
>
> Obviously in the above example - we get a log message per input file, whi=
ch
> we don't want.
>
> Another idea was to use SEDA queues and an aggregator. =A0This works - bu=
t it
> relies on knowing how many input files there were for the batchSize() in =
the
> aggregator. =A0It also introduces the batchtimeout as well. =A0It is not =
ideal -
> so I am wondering if there is a better way.
>
> from("file://trigger")
> .process(getListFilesInDirectoryProcessor())
> .split(body())
> .to("seda:atomicFiles");
>
> from("seda:atomicFiles")
> .threads(10)
> .process(getFileSplitProcessor())
> .log("File thread completed")
> .to("seda:aggregate");
>
> from("seda:aggregate")
> .aggregate(header("extractId"))
> =A0.batchSize(2) =A0// this needs to be the same as the number of input f=
iles
> =A0.batchTimeout(10000000)
> =A0.log("Processing of input files complete.");
>
> I have tried to get my head around the new async package in Camel 2.x - b=
ut
> I am having trouble understanding how to get it to do a request/reply.
>
> Is there an easier way to do what I want? =A0The Async documentation goes=
 into
> detail on how to use callbacks - none of which seem to be in DSL routes.
> The threads() function seems to be the DSL version...but it doesn't descr=
ibe
> how to do the request/reply. =A0The example route just splits to multiple
> threads and its done.
>
> What I am doing seems to be a common problem - get 1 input message...spli=
t
> the work into smaller chunks to be processed simultaneously and then when
> all pieces are done - continue on. =A0So this EIP:
> http://camel.apache.org/composed-message-processor.html
>
> The difference here is that I want the same processing to occur..just
> multi-threaded. Whereas that EIP is more for different processing on each
> piece.
>
> Thanks!
>
> Jonathan
> --
> View this message in context: http://old.nabble.com/Split-an-exchange%2C-=
process-pieces-in-parallel-and-then-wait-for-all-to-complete-tp28824028p288=
24028.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--=20
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus