camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ayush Dixit (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-11104) Performance issue in camel sftp
Date Tue, 04 Apr 2017 07:45:41 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954732#comment-15954732
] 

Ayush Dixit commented on CAMEL-11104:
-------------------------------------

Already put on user mailing list

> Performance issue in camel sftp
> -------------------------------
>
>                 Key: CAMEL-11104
>                 URL: https://issues.apache.org/jira/browse/CAMEL-11104
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-sftp
>    Affects Versions: 2.15.2
>         Environment: Performance Enviroment
>            Reporter: Ayush Dixit
>
> We have implemented a camel route where we are having camel sftp producer to transfer
a files to remote SFTP location but on performance testing on client environment and on our
local environment we have observed degradation in the time for transferring files to remote
SFTP location. Please find the detailed analysis below. The we tried the various test in our
local environment. In each test we put around 22 files on camel file consumer and each file
took below time to write the file. PFB details
> ========================================================================================
> When target directory having 20,000 files. Camel sftp producer took around 1 minute 43
second to a transfer file 
> ========================================================================================
> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt]
using exchange: Exchange[ST30PERFORPMAJAR17020726.txt]
>            INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file : ST30PERFORPMAJAR17020726.txt from IOP FTP directory
>             INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file : ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
>             DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[ST30PERFORPMAJAR17020726.txt]
>             DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFORPMAJAR17020726.txt
(no charset)
>            DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file: QST30PERFORPMAJAR17020726.txt
using stream: java.io.BufferedInputStream@54a89ff5
>            DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.658 seconds (658 millis)
to store file: QST30PERFORPMAJAR17020726.txt and FTP client returned: true
>            DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QST30PERFORPMAJAR17020726.txt]
to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
>            DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QST30PERFORPMAJAR17020726.txt
to: icoms/1/ST30PERFORPMAJAR17020726.txt
>             INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : ST30PERFORPMAJAR17020726.txt successfully sent to ICOMS FTP directory
> =======================================================================================
> When target directory having 40,000 files. Camel sftp producer took around 3 minute 17
second to transfer file
> =======================================================================================
> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.FileConsumer> About to process file: GenericFile[/data/users/slvm02/smp53/domains/v
>                                 mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt] using
exchange: Exchange[SX30RPMAJAR1702483756.txt]
>                 INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Picked IOP response file :
>                                 SX30RPMAJAR1702483756.txt from IOP FTP directory
>                 INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
Sending IOP response file
>                                 : SX30RPMAJAR1702483756.txt to ICOMS FTP directory
>                 DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPer
>                                 Poll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]
Exchange[SX30RPMAJAR1702483756.txt]
>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.GenericFileConverter> Read file /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt
(no charset)
>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> About to store file: QSX30RPMAJAR1702483756.txt
using stream: java.io.BufferedInputStream@7954cfa8
>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Took 0.659 seconds (659 millis)
to store file: QSX30RPMAJAR1702483756.txt and FTP client returned: true
>                 DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.RemoteFileProducer> Wrote [icoms/1/QSX30RPMAJAR1702483756.txt]
to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser]]
>                 DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
org.apache.camel.component.file.remote.SftpOperations> Renaming file: icoms/1/QSX30RPMAJAR1702483756.txt
to: icoms/1/SX30RPMAJAR1702483756.txt
>                 INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 - file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResponseFtpRouteBuilder>
IOP response file : SX30RPMAJAR1702483756.txt successfully sent to ICOMS FTP directory
> ====================================================================================================
> Similarly when we achieved the files from target directory. It took around 6 sec.It seems
like there is a performance issue with camel sftp component. Does it list the files in target
directory which is taking time.
> ===================================================================================================
> PFB the producer route which we set up
> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message