camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laurentiu Trica <laurentiu.tr...@finalfolder.biz>
Subject Re: FTP2 Component fails to process multiple files at once
Date Tue, 24 May 2011 15:18:18 GMT
Hello,

I added a bug on JIRA: https://issues.apache.org/jira/browse/CAMEL-4010

I think the problem only occurs when the path to FTP includes a folder -
this means your files are not on the root of the FTP account.

On Tue, May 24, 2011 at 1:32 PM, Laurentiu Trica <
laurentiu.trica@finalfolder.biz> wrote:

> Hi,
>
> The problem is not that concurrent FTP components are accessing the same
> files.
> The problem is that the one FTP component which handles the files from the
> FTP account is not doing it in separate connections.
> And this is a bad thing to do in a very delicate folder based environment
> like a FTP account.
>
> As you can see from the FTP server logs, there is only one FTP consumer
> connection which handles all the files it finds. The problem is that
> handling the files concurrently on a single FTP connection makes the output
> unpredictable.
>
> I can send my test. Where should I put it? What endpoints should I use? Do
> you have a public access ftp account to test it?
>
>
> On Mon, May 23, 2011 at 7:13 PM, Taariq Levack <taariql@gmail.com> wrote:
>
>> Hi
>>
>> I don't know if it's a bug or not, but I would expect it to work.
>> There's an old ticket to make FTP threadsafe[1], but I only see a
>> concurrent
>> producer test.
>>
>> Can you post your test?
>>
>> [1]https://issues.apache.org/jira/browse/CAMEL-1641
>>
>>
>> On Mon, May 23, 2011 at 11:12 AM, Laurentiu Trica <
>> laurentiu.trica@finalfolder.biz> wrote:
>>
>> > Hello again,
>> >
>> > Anyone had this issue before? Is it a bug in Camel FTP or it's just the
>> way
>> > it is supposed to work?
>> >
>> > On Fri, May 20, 2011 at 6:00 PM, Laurentiu Trica <
>> > laurentiu.trica@finalfolder.biz> wrote:
>> >
>> > > Hello,
>> > >
>> > > I've been struggling to solve a situation in which a FTP poller was
>> not
>> > > working from time to time.
>> > > I've got to the point where I think I know what the problem is.
>> > >
>> > > I'm using Camel 2.6 in servicemix.
>> > >
>> > > *I've put up a test for this:*
>> > > - three FTP file producers at different repeating period of time: 35s,
>> > > 5min, 1h. Note that when starting servicemix they all put the first
>> file
>> > at
>> > > once.
>> > > - the files ar put on the FTP in /in folder
>> > > - one FTP (ftps) consumer which deletes the files after consuming them
>> > > - the route is simply moving the file from input (ftps) to output (a
>> > local
>> > > folder).
>> > >
>> > > *The problem*:
>> > > - only the first produced file is consumed but is processed over and
>> over
>> > > again
>> > > - the rest of the files (the rest of the first 3 files + the others
>> that
>> > > are added every 35s) are just ignored
>> > >
>> > > Possible problem:
>> > > The FTP component is getting the list of the first 3 files from the
>> FTP
>> > in
>> > > a threaded manner which depends on the state of the current directory
>> on
>> > the
>> > > FTP server
>> > > - it changes the current directory to /in
>> > > - retrieves the file
>> > > - it changes the current directory to /
>> > > - the second (probably) thread changes the directory to /in
>> > > - retreives the second file
>> > > - the first (probably) thread sends a delete command but it sends it
>> as
>> > if
>> > > it were in the root folder but it is actually in the /in folder
>> > > - the FTP gives an error that the file doesn't exist
>> > > - somehow the FTP Component blocks itself on the first file...
>> > >
>> > > You can see the partial (useful) log below:
>> > >
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 257 "/" is
>> > > current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> CWD in
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 250 CWD
>> > > successful. "/in" is current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> PASV
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 227
>> Entering
>> > > Passive Mode (192,168,0,25,228,206)
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> RETR
>> > > WS_1001_1.xml
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 150
>> Connection
>> > > accepted
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> SSL
>> connection
>> > > for data connection established
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 226
>> Transfer
>> > OK
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> CWD /
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 250 CWD
>> > > successful. "/" is current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> PWD
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 257 "/" is
>> > > current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> CWD in
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 250 CWD
>> > > successful. "/in" is current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> PASV
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 227
>> Entering
>> > > Passive Mode (192,168,0,25,228,207)
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> RETR
>> > > WS_1001_2.xml
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 150
>> Connection
>> > > accepted
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> DELE
>> > > in/WS_1001_1.xml
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 550 File
>> not
>> > > found
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> PASV
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 227
>> Entering
>> > > Passive Mode (192,168,0,25,228,208)
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> NLST in
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 550
>> Directory
>> > > not found
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> PWD
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 257 "/in"
>> is
>> > > current directory.
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> CWD in
>> > > (002386)5/20/2011 17:07:03 PM - sftp-test (192.168.0.194)> 550 CWD
>> > failed.
>> > > "/in/in": directory not found.
>> > >
>> > > For the whole logs go here:
>> > > http://pastie.org/private/kknnkxpbyzqmai7gsm8z5g
>> > > For the Servicemix log:
>> http://pastie.org/private/eulookxiepan3nkjyu41a
>> > >
>> > > A workaround that seems to work is to use the maxMessagesPerPoll=1
>> > > option...
>> > >
>> > > Anyone has the same problem? Can this be fixed?
>> > >
>> > > Thank you for your time. Enjoy the weekend!
>> > >
>> > > Laurentiu Trica
>> > >
>> >
>> >
>> >
>> > --
>> > Laurentiu Trica
>> > Software Developer             Mobile: (+40) 722 329318
>> > S.C MoreDevs S.R.L.          Email: laurentiu.trica@finalfolder.biz
>> >
>> > This message can contain privileged or confidential information and it
>> is
>> > intended only for addressee. Any unauthorized disclosure is strictly
>> > prohibited.
>> >
>>
>
>
>
> --
> Laurentiu Trica
> Software Developer             Mobile: (+40) 722 329318
> S.C MoreDevs S.R.L.          Email: laurentiu.trica@finalfolder.biz
>
> This message can contain privileged or confidential information and it is
> intended only for addressee. Any unauthorized disclosure is strictly
> prohibited.
>



-- 
Laurentiu Trica
Software Developer             Mobile: (+40) 722 329318
S.C MoreDevs S.R.L.          Email: laurentiu.trica@finalfolder.biz

This message can contain privileged or confidential information and it is
intended only for addressee. Any unauthorized disclosure is strictly
prohibited.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message