camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Fetching two files from FTP at the same time
Date Thu, 20 Sep 2012 10:46:41 GMT
On Thu, Sep 20, 2012 at 12:41 PM, Bilgin Ibryam <bibryam@gmail.com> wrote:
> Unfortunately we cannot open more than one connection at time, so it is not
> possible to have two consumers reading the two (actually now it is three)
> files at the same time.
>
> IMHO this sounds a bit odd/dangerous way of transferring data between
>> parties.
>> There is no good way to tell if these 2 files are in sync, or the
>> first file has new content, and the 2nd has previous content.
>>
>> It would be better if the other party could change its strategy to
>> upload new files using a name pattern to pair the files, eg
>> a sequence number that would match etc.
>>
>> myfile-123.dat
>> myotherfile-123.dat
>>
>> Or to use a 3rd "done" file to tell you that the 2 files has now been
>> properly written. eg deleting the done file first, write the 2 files,
>> and save the done file again.
>>
>
> Sequencing would be great, but we will not have it.
>
>
>
>>
>>
>> Can you move the files before you download them? As a move operation
>> on a remote FTP server is faster than having to download the 2 files
>> in parallel in hopes that you make it before the other party starts to
>> override the files.
>>
>> If you can move the files (or rename) then thats likely a near atomic
>> IO operation, and thus would be fast doing 2 file moves. Although you
>> would need to send 2 FTP commands.
>>
>
>
> No, we cannot move or rename the files, because they are consumed also by
> other parties, we don't know about.
>
>
>>
>> Camel has a preMove option, but its still based on the
>> one-file-at-a-time strategy the ftp consumer uses.
>>
>>
>> If you know the file names in advance, then you can have 2 routes,
>> each route picking up the designated file. And if you are allowed to
>> delete the file after download, then you could assume if there is a
>> new file then its new content since last.
>>
>> But the 2 sec frequency is a bit fast, which mean you would need to
>> download and delete the file < 2s. And have the polling frequency of
>> the ftp consumer aligned with the other party.
>>
>
> Can you tell me what is 2 sec frequency, where?
>

You mentioned that the files will be overwritten with a 2 second frequency.

== quote ===
The reason for that is there is another process which writes to the ftp
folder and replaces old files with new ones every couple of seconds.
== quote ===


>
> What would be ideal is, if we able to lock the files we want to read before
> we start downloading them.
>
> For example open the two files, and then start the download. The only
> guarantee we have is that: a file will not be overridden while we are
> reading it.
>

And how is that possible? Locking files over FTP is "hard".


> Do you thing it is possible to extend the component, so that it opens more
> than one file, and then start reading them?
>

No thats not easy the ftp client dont support multi threading.

And as well the file/ftp component is based on polling one file at a time.

You have a very special use-case. And its possible better to write
your own FTP logic to handle that.


>
> Bilgin
>
>
>>
>>
>>
>> > Thanks
>> > Bilgin
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

Mime
View raw message