camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayush Dixit <Ayush.Di...@sigma-systems.com>
Subject Performance issue in Camel SFTP component
Date Tue, 04 Apr 2017 07:23:03 GMT


Hi ,


We have implemented a camel route where we are having camel sftp producer to transfer a files
to remote SFTP location but on performance testing on client environment and on our local
environment we have observed degradation in the time for transferring files to remote SFTP
location. Please find the detailed analysis below. The we tried the various test in our local
environment. In each test we put around 22 files on camel file consumer and each file took
below time to write the file. PFB details


========================================================================================
When target directory having 20,000 files. Camel sftp producer took around 1 minute 43 second
to a transfer file
========================================================================================

DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt]
using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
           INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
            INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
            DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[ST30PERFORPMAJAR17020726.txt]



            DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt
(no charset)
           DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> About to store file: QST30PERFORPMAJAR17020726.txt
using stream: java.io.BufferedInputStream@54a89ff5<mailto:java.io.BufferedInputStream@54a89ff5>
           DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658 millis)
to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned: true

           DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QST30PERFORPMAJAR17020726.txt]
to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
           DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QST30PERFORPMAJAR17020726.txt
to: icoms/1/ST30PERFORPMAJAR17020726.txt
            INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS FTP directory


=======================================================================================
When target directory having 40,000 files. Camel sftp producer took around 3 minute 17 second
to transfer file
=======================================================================================

DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/v
                                mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using
exchange: Exchange[SX30RPMAJAR1702483756.txt]
                INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file :
                                SX30RPMAJAR1702483756.txt from IOP FTP directory
                INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file
                                : SX30RPMAJAR1702483756.txt to ICOMS FTP directory
                DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
                                Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[SX30RPMAJAR1702483756.txt]


                DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt
(no charset)
                DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> About to store file: QSX30RPMAJAR1702483756.txt
using stream: java.io.BufferedInputStream@7954cfa8<mailto:java.io.BufferedInputStream@7954cfa8>
                DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659 millis)
to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
                DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QSX30RPMAJAR1702483756.txt]
to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
                DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QSX30RPMAJAR1702483756.txt
to: icoms/1/SX30RPMAJAR1702483756.txt
                INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1<file:///\\data\users\slvm02\smp53\domains\vmb\Icoms\iop_responses\1>)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP directory



====================================================================================================
Similarly when we achieved the files from target directory. It took around 6 sec.It seems
like there is a performance issue with camel sftp component. Does it list the files in target
directory which is taking time.
===================================================================================================

PFB the producer route which we set up
sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser


Ayush Dixit
Software Engineer
ayush.dixit@sigma-systems.com<mailto:ayush.dixit@sigma-systems.com>
O: +91.20.25661992 x 321 |  M: +91-814-982-6260 | Skype: ayushdixit421
[cid:image001.jpg@01D16809.E12C0D50]<http://www.sigma-systems.com/>




Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message