camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Hoffer <dhoff...@gmail.com>
Subject sftp endpoint is not as performant as expected
Date Wed, 11 Nov 2015 18:31:03 GMT
I have a spring configured (XML) Camel project where where are many routes
but the starting point of all the data is an sftp URI.  E.g.

<from uri="sftp://{{gp.camel.sg.username}}@
{{gp.camel.sg.host}}:{{gp.camel.sg.port}}/{{gp.camel.sg.path}}?password={{gp.camel.sg.password}}&amp;delete=true&amp;exclusiveReadLockStrategy=#nonBlockingSftpReadLockStrategy&amp;readLockCheckInterval=60000&amp;readLockTimeout=360000000&amp;filter=#sgHiFilter&amp;reconnectDelay=30000&amp;delay=60000"/>
<to
uri="file://{{gp.home}}/work/sg_decrypt/?tempFileName=${file:name}.partial"/>

There are several more subsequent routes that are all file based except the
very last one which calls a bean URI (web service call)

This mostly works fine but what is happening is that Camel is not pulling
the data from the SFTP server fast enough.  At times the SFTP server can
get a lot of files...say about 1000.  We want those to be consumed by this
process as soon as they are available on the SFTP server.  The sftp route
does check via the nonBlockingSftpReadLockStrategy bean that the file has
not changed since the last time it was identified (e.g. second poll of file
will return true for acquireExclusiveReadLock() if last modified date and
file size has not changed.  And the poll frequency is 60 seconds so we
expect the file to be on the SFTP server for 2-3 minutes then be sent to
the next route.

What is happening is that when the SFTP server receives a lot of files the
Camel route slows down and it only processes the files very slowly.  I'm
trying to understand why it would slow down.

Btw, we can use mget to quickly get all the files from the SFTP server and
put them in the same folder that the Camel route uses as its destination
and that works fine.  So why can't the sftp endpoint do the same thing?

Also, we don't have any thread pools configured for these routes except for
one where we have a muticast route configured for parallel processing.  So
all the rest should be using the default Camel thread pool.  I did read
online that the sftp input endpoint is single threaded so I'm wondering if
that has something to do with the problem?  But not sure how.  We do want
maximum concurrency with all the routes.

-Dave

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message