camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Parallel processing of File Component on clustered environment
Date Tue, 12 Nov 2013 16:31:11 GMT
Hi

You can use threads in your route to hand over the processing to a
thread pool, then the file consumer can pickup the next file.

I wrote an old blog about  that
http://www.davsclaus.com/2009/05/on-road-to-camel-20-concurrency-with.html

Though back then the we named it async, but its threads.



On Tue, Nov 12, 2013 at 5:09 PM, wiseotherwise <munishekarraja@gmail.com> wrote:
> Hi,
> I am looking for someone to help me out on this issue.
>
> Our implementation is to read files placed in an INPUT folder and convert
> those to respective XMLs on OUTPUT folder. We have 2 instances of TOMCAT
> server running to listen to this INPUT Folder for load balancing and
> parallel processing.
>
>  <from
> uri="file://{{initial.input.dir.path}}?move={{archive.dir.path}}/inbox/initial&amp;maxMessagesPerPoll=5&amp;delay=10000&amp;readLock=changed&amp;readLockTimeout=5000&amp;readLockLoggingLevel=INFO"
> />
>
> The issue we are facing is, even though the polling size is 5 we have only 2
> files that are getting picked and processed. For Example: File1.csv and
> File2.csv are picked in the first polling. If File2.csv is processed in 2
> min and lets say File1.csv is taking a long time(20 min). Once the File2.csv
> is processed, CAMEL is not picking the File3.csv. Its waiting all the way
> until File1.csv is processed and then picks only File3.csv (Where as it
> should pick 2 files as there are 2 instances).
>
> What we want to achieve is: We have 6 instances on higher environment and we
> want all the instances to pick files and process them parallel so that it
> wont pool one and the other wait to finish.
>
> Appreciate your help.
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Parallel-processing-of-File-Component-on-clustered-environment-tp5743115.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message