camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Capwell <dcapw...@gmail.com>
Subject Re: unable to control ftp delay
Date Thu, 28 Jun 2012 22:32:47 GMT
Playing with the delay fields defined in file2 seems to improve download
speeds.  Now that this is working better I want to be able to skip any file
that has already been processed (should persist on restart).  It looked to
me that I should be able to set this up by using the filter option, but I
am having a hard time figuring out how to use this option without using
spring.

here is the code I am using (might not compile cause its copy/paste from
diff places):

SimpleRegistry registry = new SimpleRegistry();
registry.put("fileExistsFilter", new CopyFilter());

CamelContext context = new DefaultCamelContext(registry);
context.addRoutes(new RouteBuilder() {
      @Override
      public void configure() throws Exception {
          from("sftp://example.com/path/to/file?password=" + pass +
"&binary=true&delay="+delay+ "&initialDelay=" + delay +
"&filter=#fileExistsFilter")
            .to("file://data/ftp?fileExist=Ignore");

      }
    });
context.start();

...
public static class CopyFilter implements GenericFileFilter {

    @Override
    public boolean accept(GenericFile genericFile) {
      // check to see if file exists locally
      File localFile = new File("data/ftp", genericFile.getFileName());
      if(localFile.exists()) {
        // check to make sure the sizes match
        if(localFile.length() == genericFile.getFileLength()) {
          // file has already been copied, ignore
          return false;
        }
      }
      return true;
    }
  }


Thanks for your time reading this email

On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapwell@gmail.com> wrote:

> I'll take a look at file2 to see if that helps.
>
> My goal is to push the data from FTP to s3.  Copying locally so I know
> what has been processed,
> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <claus.ibsen@gmail.com> wrote:
>
>> Hi
>>
>> As mentioned on the ftp2 page
>> http://camel.apache.org/ftp2
>>
>> See the file2 page for more options as the ftp component inherits these
>> options
>> http://camel.apache.org/file2.html
>>
>> The delay and consumer.delay option is the same. delay is just
>> shorthand for consumer.delay.
>>
>> And no you cannot avoid the download if you later want to upload it
>> someplace else, and the file exists.
>> The ftp consumer don't know about this.
>>
>> A tricky improvement could be to add a new option to only download the
>> file on demand, but that is a bit tricky
>> as you would need a live connection and the ftp client to still be
>> around and connected.
>>
>>
>> What are you trying to do? To sync files between 2 ftp servers?
>>
>>
>>
>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapwell@gmail.com>
>> wrote:
>> > I am trying to use sftp (camel 2.9.2) and notice that every download is
>> > spread out to one every 10 seconds (files are around 10kb).  Based off
>> the
>> > examples in http://camel.apache.org/ftp2.html it looks like there are
>> delay
>> > options (they are not listened under options, but used in example) but
>> when
>> > I use them nothing changes.  How can I have the client pull files
>> faster?
>> >
>> > Another question, if I am trying to write the ftp files locally, is
>> there
>> > any way to avoid the file download if the local file is around?
>> >  I append fileExist=Ignore in the to() but it seems that the files still
>> > get written again.
>> >
>> > Here is the route I am using:
>> >
>> > public void ftpToLocal() throws Exception {
>> >    runRoutes(new RouteBuilder() {
>> >      @Override
>> >      public void configure() throws Exception {
>> >        Registry registry = getContext().getRegistry();
>> >        String user = (String) registry.lookup("ftpUser");
>> >        String pass = (String) registry.lookup("ftpPass");
>> >
>> >        // we use a delay of 60 minutes (eg. once pr. hour we poll the
>> FTP
>> > server
>> > //        long delay = 60 * 60 * 1000L;
>> > //        long delay = TimeUnit.SECONDS.toMillis(1);
>> >        long delay = 200;
>> >
>> >        from("sftp://"+user+"@example.com/path/to/dir?password=" + pass
>> +
>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay)
>> >            .to("file://data/ftp?fileExist=Ignore");
>> >      }
>> >    });
>> >  }
>> >
>> >
>> > Thanks for your time reading this email.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message