camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <>
Subject Re: Parallel processing/multiple file consumer threads with readLock=changed and long timeout?
Date Tue, 02 Jun 2015 07:11:44 GMT

If possible then using done file names is IMHO a better strategy.
Though that would require the other party to do this when it uploads
to the FTP server.

On Mon, Jun 1, 2015 at 6:10 PM, Tom Duncalf <> wrote:
> Hi,
> I am building a Camel route to consume files uploaded by FTP and then upload
> them elsewhere. It would seem that "changed" is the most suitable readLock
> strategy for this, and I would like to set a fairly large
> readLockCheckInterval and readLockTimeout (e.g. 20s and 40s) to help prevent
> incomplete files from users with slow/intermittent connections being picked
> up by the route prematurely.
> What I would like to achieve is to make it so that the File consumer is not
> blocked from picking up more files for the duration of the readLockTimeout -
> that is, currently, if file1 is uploaded at t=0 and file2 uploaded at t=5,
> then the consumer picks up file1 at t=0 and is then blocked until at least
> t=20 (assuming a 20s check interval), so file2 will not be picked up until
> at least t=20 (and processed until at least t=40). Instead, I would like
> there to be (for example) more than one consumer thread, so that when the
> first consumer thread picks up file1 and is waiting until t=20, another
> consumer thread can still pick up file2 at t=5 and wait until t=25.
> None of the concurrency options available seem to cover this scenario - I
> understand that I can decouple the file being consumed and the subsequent
> processing using, for example, a SEDA queue, or I can split the processing
> into multiple threads after a file has been processed by the File consumer
> using the Threads DSL, but I can't see how to make the consumer itself
> consume files and therefore create messages in a multi-threaded manner.
> One workaround may be to spawn a new process for each FTP user and have it
> consume from their home directory, but I would prefer to avoid this
> additional complexity and it would not gracefully handle a situation where
> one user uploads a large volume of files. One other option I can think of is
> to start multiple instances of my route somehow (to create multiple
> consumers), and then synchronise them with a shared repository for the
> inProgressRepository, but I'm not sure if that would actually work.
> Any input would be great - even if it is basic as I am new to Camel!
> Thanks,
> Tom
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

Claus Ibsen
Red Hat, Inc.
Twitter: davsclaus
Author of Camel in Action:

View raw message