camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Hoffer <>
Subject Re: sftp endpoint is not as performant as expected
Date Thu, 12 Nov 2015 21:05:24 GMT
I'm rather new to Camel, I'm assuming that if the SFTP source gets 900
files then the configured
GenericFileExclusiveReadLockStrategy#acquireExclusiveReadLock will get
called 900 times (once per file) every polling cycle.

We too have a custom GenericFileExclusiveReadLockStrategy class that does
not block.  What ours does is maintain a map/cache of of the
lastModifiedTime and fileSize for each of the 900 files.  The call to
acquireExclusiveReadLock() will return true as soon as neither of those
values changed since the last poll cycle.

We have our delay/poll cycle set to 60 seconds.  I'm not sure why such a
large value but we are fine waiting 2 minutes to get each of the 900
files...and by the time 2 minutes go by we will likely have more files in
the SFTP source.  E.g. if we have a constant delay in receiving each file
of 2-3 minutes that is more than fine.  The problem is that on each cycle
it only processes 3-5 files instead of the 900 that should have returned
true from acquireExclusiveReadLock()

What seems to be happening is that Camel is not calling the
acquireExclusiveReadLock() method for each of the 900 files every 60
seconds, rather it slows down and either calls just a few of those 900
files or none.

Any ideas?


Btw, I'm  using Camel 2.8.2

On Thu, Nov 12, 2015 at 12:43 PM, pmmerritt <> wrote:

> We had a similar issue and traced it down to our usage of
> SftpChangedExclusiveReadLockStrategy which has a default check interval of
> 5
> seconds which causes the polling to slow down waiting on files to finish
> being written, so max you can do is about 1 per 5 seconds. You can change
> the checkInterval so that it is faster, or we ended up writing our own read
> lock strategy that did not block the polling thread
> --
> View this message in context:
> Sent from the Camel - Users mailing list archive at

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message