camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <>
Subject Re: Parallel processing of File Component on clustered environment
Date Tue, 12 Nov 2013 16:31:11 GMT

You can use threads in your route to hand over the processing to a
thread pool, then the file consumer can pickup the next file.

I wrote an old blog about  that

Though back then the we named it async, but its threads.

On Tue, Nov 12, 2013 at 5:09 PM, wiseotherwise <> wrote:
> Hi,
> I am looking for someone to help me out on this issue.
> Our implementation is to read files placed in an INPUT folder and convert
> those to respective XMLs on OUTPUT folder. We have 2 instances of TOMCAT
> server running to listen to this INPUT Folder for load balancing and
> parallel processing.
>  <from
> uri="file://{{initial.input.dir.path}}?move={{archive.dir.path}}/inbox/initial&amp;maxMessagesPerPoll=5&amp;delay=10000&amp;readLock=changed&amp;readLockTimeout=5000&amp;readLockLoggingLevel=INFO"
> />
> The issue we are facing is, even though the polling size is 5 we have only 2
> files that are getting picked and processed. For Example: File1.csv and
> File2.csv are picked in the first polling. If File2.csv is processed in 2
> min and lets say File1.csv is taking a long time(20 min). Once the File2.csv
> is processed, CAMEL is not picking the File3.csv. Its waiting all the way
> until File1.csv is processed and then picks only File3.csv (Where as it
> should pick 2 files as there are 2 instances).
> What we want to achieve is: We have 6 instances on higher environment and we
> want all the instances to pick files and process them parallel so that it
> wont pool one and the other wait to finish.
> Appreciate your help.
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

Claus Ibsen
Red Hat, Inc.
Twitter: davsclaus
Author of Camel in Action:

View raw message