camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ayush Dixit <Ayush.Di...@sigma-systems.com>
Subject RE: Performance issue in Camel SFTP component
Date Thu, 06 Apr 2017 05:30:49 GMT
Should I raise a JIRA for that?

-----Original Message-----
From: Ayush Dixit 
Sent: Wednesday, April 5, 2017 1:22 PM
To: users@camel.apache.org
Subject: RE: Performance issue in Camel SFTP component

Hi Claus,

Already tried that , See below endpoint configuration . No luck :( 

to("sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser&stepwise=false")

Thanks
Ayush 

-----Original Message-----
From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
Sent: Wednesday, April 5, 2017 1:08 PM
To: users@camel.apache.org
Subject: Re: Performance issue in Camel SFTP component

Try with maxMessagesPerPoll to set an upper limit.

On Tue, Apr 4, 2017 at 12:10 PM, Ayush Dixit <Ayush.Dixit@sigma-systems.com> wrote:
>
> Thanks Claus , I've tried  stepwise=false  in camel producer . No such 
> luck
>
> Also, I have tried download=false and useList=false. Still the 
> performance is same. I am referring the below link for camel 
> documentation
>
> http://camel.apache.org/ftp2.html
>
> Is there is anything else you think I can try?
>
> Thanks
> Ayush
>
>
> -----Original Message-----
> From: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Sent: Tuesday, April 4, 2017 1:35 PM
> To: users@camel.apache.org
> Cc: users-subscribe@camel.apache.org
> Subject: Re: Performance issue in Camel SFTP component
>
> Please spend more time to read the documentation and you can find 
> options to tweak the options to use or not use FTP list etc, and there 
> is also a stepwise option
>
> On Tue, Apr 4, 2017 at 9:16 AM, Ayush Dixit 
> <Ayush.Dixit@sigma-systems.com>
> wrote:
>
>>
>>
>> Hi ,
>>
>>
>>
>>
>>
>> We have implemented a camel route where we are having camel sftp 
>> producer to transfer a files to remote SFTP location but on 
>> performance testing on client environment and on our local 
>> environment we have observed degradation in the time for transferring files to remote
SFTP location.
>> Please find the detailed analysis below. The we tried the various 
>> test in our local environment. In each test we put around 22 files on 
>> camel file consumer and each file took below time to write the file.
>> PFB details
>>
>>
>>
>>
>>
>> ============================================================
>> ============================
>>
>> When target directory having 20,000 files. Camel sftp producer took 
>> around
>> 1 minute 43 second to a transfer file
>>
>> ============================================================
>> ============================
>>
>>
>>
>> DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.FileConsumer> About to process file:
>> GenericFile[/data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/
>> 1 /ST30PERFORPMAJAR17020726.txt] using exchange:
>> Exchange[ST30PERFORPMAJAR17020726.txt]
>>
>>            INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> Picked IOP response file :
>> ST30PERFORPMAJAR17020726.txt from IOP FTP directory
>>
>>             INFO 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> Sending IOP response file :
>> ST30PERFORPMAJAR17020726.txt to ICOMS FTP directory
>>
>>             DEBUG 07:00:38 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.
>> 190/icoms/1?download=false&maxMessagesPerPoll=10&
>> password=xxxxxx&tempPrefix=Q&username=sigmauser] Exchange[ 
>> ST30PERFORPMAJAR17020726.txt]
>>
>>
>>
>>
>>
>>
>>
>>             DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.GenericFileConverter> Read file 
>> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/ST30PERFOR
>> P
>> MAJAR17020726.txt
>> (no charset)
>>
>>            DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> About to store
>> file: QST30PERFORPMAJAR17020726.txt using stream:
>> java.io.BufferedInputStream@54a89ff5
>>
>>            DEBUG 07:02:19 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Took 0.658 
>> seconds
>> (658 millis) to store file: QST30PERFORPMAJAR17020726.txt and FTP 
>> client
>> returned: true
>>
>>
>>
>>            DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote 
>> [icoms/1/QST30PERFORPMAJAR17020726.txt]
>> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&
>> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
>> ]
>> ]
>>
>>            DEBUG 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
>> icoms/1/QST30PERFORPMAJAR17020726.txt to: icoms/1/ 
>> ST30PERFORPMAJAR17020726.txt
>>
>>             INFO 07:02:20 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> IOP response file : ST30PERFORPMAJAR17020726.txt
>> successfully sent to ICOMS FTP directory
>>
>>
>>
>>
>>
>> ============================================================
>> ===========================
>>
>> When target directory having 40,000 files. Camel sftp producer took 
>> around
>> 3 minute 17 second to transfer file
>>
>> ============================================================
>> ===========================
>>
>>
>>
>> DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.FileConsumer> About to process file:
>> GenericFile[/data/users/slvm02/smp53/domains/v
>>
>>
>> mb/Icoms/iop_responses/1/SX30RPMAJAR1702483756.txt]
>> using exchange: Exchange[SX30RPMAJAR1702483756.txt]
>>
>>                 INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder>
>> Picked IOP response file :
>>
>>                                 SX30RPMAJAR1702483756.txt from IOP 
>> FTP directory
>>
>>                 INFO 07:47:23 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder>
>> Sending IOP response file
>>
>>                                 : SX30RPMAJAR1702483756.txt to ICOMS 
>> FTP directory
>>
>>                 DEBUG 07:47:23 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.processor.SendProcessor> >>>> Endpoint[sftp://10.100.150.
>> 190/icoms/1?download=false&maxMessagesPer
>>
>>                                 Poll=10&password=xxxxxx& 
>> tempPrefix=Q&username=sigmauser] Exchange[SX30RPMAJAR1702483756.txt]
>>
>>
>>
>>
>>
>>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.GenericFileConverter> Read file 
>> /data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1/SX30RPMAJA
>> R
>> 1702483756.txt
>> (no charset)
>>
>>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> About to store
>> file: QSX30RPMAJAR1702483756.txt using stream:
>> java.io.BufferedInputStream@
>> 7954cfa8
>>
>>                 DEBUG 07:50:40 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Took 0.659 
>> seconds
>> (659 millis) to store file: QSX30RPMAJAR1702483756.txt and FTP client
>> returned: true
>>
>>                 DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.RemoteFileProducer> Wrote 
>> [icoms/1/QSX30RPMAJAR1702483756.txt]
>> to [Endpoint[sftp://10.100.150.190/icoms/1?download=false&
>> maxMessagesPerPoll=10&password=xxxxxx&tempPrefix=Q&username=sigmauser
>> ]
>> ]
>>
>>                 DEBUG 07:50:41 (Camel (cfgMgrCamelContext) thread #6
>> -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> org.apache.camel.component.file.remote.SftpOperations> Renaming file:
>> icoms/1/QSX30RPMAJAR1702483756.txt to:
>> icoms/1/SX30RPMAJAR1702483756.txt
>>
>>                 INFO 07:50:41 (Camel (cfgMgrCamelContext) thread #6 -
>> file:///data/users/slvm02/smp53/domains/vmb/Icoms/iop_responses/1)
>> com.sigma.samp.imp.virginmedia.bss.voice.camelroutes.iopRoute.IOPResp
>> o
>> nseFtpRouteBuilder> IOP response file : SX30RPMAJAR1702483756.txt
>> successfully sent to ICOMS FTP directory
>>
>>
>>
>>
>>
>>
>>
>> ============================================================
>> ========================================
>>
>> Similarly when we achieved the files from target directory. It took 
>> around
>> 6 sec.It seems like there is a performance issue with camel sftp component.
>> Does it list the files in target directory which is taking time.
>>
>> ============================================================
>> =======================================
>>
>>
>>
>> PFB the producer route which we set up
>>
>> sftp://10.100.150.190/icoms/1?download=false&maxMessagesPerPoll=10&
>> password=xxxxxx&tempPrefix=Q&username=sigmauser
>>
>>
>>
>>
>>
>> Please let us know if you need anything else.
>>
>>
>>
>>
>>
>> *Ayush Dixit*
>> *Software Engineer*
>> ayush.dixit@sigma-systems.com
>>
>> O: +91.20.25661992 x 321 <+91%2020%202566%201992> |  M:
>> +91-814-982-6260 <+91%2081498%2026260> | Skype: ayushdixit421
>>
>> [image: cid:image001.jpg@01D16809.E12C0D50]
>> <http://www.sigma-systems.com/>
>>
>>
>>
>>
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Mime
View raw message