camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Camel > FTP
Date Mon, 29 Mar 2010 01:07:00 GMT
<html>
<head>
    <base href="http://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1519/1/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background-color: white" bgcolor="white">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
     <h2><a href="http://cwiki.apache.org/confluence/display/CAMEL/FTP">FTP</a></h2>
     <h4>Page <b>edited</b> by             <a href="http://cwiki.apache.org/confluence/display/~njiang">willem
jiang</a>
    </h4>
     
          <br/>
     <div class="notificationGreySide">
         <h2><a name="FTP-FTP%2FSFTPComponentCamel1.xonly"></a>FTP/SFTP
Component - <b>Camel 1.x only</b></h2>

<p>This component provides access to remote file systems over the FTP and SFTP protocols.</p>

<p>Maven users will need to add the following dependency to their <tt>pom.xml</tt>
for this component:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
&lt;dependency&gt;
    &lt;groupId&gt;org.apache.camel&lt;/groupId&gt;
    &lt;artifactId&gt;camel-cometd&lt;/artifactId&gt;
    &lt;version&gt;x.x.x&lt;/version&gt;
    &lt;!-- use the same version as your Camel core version --&gt;
&lt;/dependency&gt;

{info:title=Using Camel 2.x}
If you are using Camel 2.x then see the FTP documentation at his [link|FTP2].
This documentation is only <span class="code-keyword">for</span> Camel 1.x.
{info} 

h3. URI format

</pre>
</div></div>
<p><a href="ftp://<span class="error">&#91;username@&#93;</span>hostname<a
href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=port&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">:port</a>/filename<a href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=%3Foptions&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">&#63;options</a>" rel="nofollow">ftp://<span class="error">&#91;username@&#93;</span>hostname<a
href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=port&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">:port</a>/filename<a href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=%3Foptions&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">&#63;options</a></a><br/>
sftp://<span class="error">&#91;username@&#93;</span>hostname<a href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=port&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">:port</a>/filename<a href="/confluence/pages/createpage.action?spaceKey=CAMEL&amp;title=%3Foptions&amp;linkCreation=true&amp;fromPageId=57232"
class="createlink">&#63;options</a></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">

Where *filename* represents the underlying file name or directory. Can contain nested folders.
The *username* is currently only possible to provide in the hostname parameter.

If no *username* is provided then {{anonymous}} login is attempted using no password.
If no *port* number is provided, Camel will provide <span class="code-keyword">default</span>
values according to the protocol (ftp = 21, sftp = 22).

You can append query options to the URI in the following format, {{?option=value&amp;option=value&amp;...}}

h3. URI Options

{div:class=confluenceTableSmall}
|| Name || Default Value || Description ||
| {{directory}} | {{<span class="code-keyword">true</span>}} | Indicates whether
or not the given file name should be interpreted by <span class="code-keyword">default</span>
as a directory or file (as it sometimes hard to be sure with some FTP servers). |
| {{password}} | {{<span class="code-keyword">null</span>}} | Specifies the password
to use to log in to the remote file system. |
| {{binary}} | {{<span class="code-keyword">false</span>}} | Specifies the file
transfer mode {{BINARY}} or {{ASCII}}. Default is {{ASCII}}. |
| {{ftpClientConfig}} | {{<span class="code-keyword">null</span>}} | Camel 1.5:
Reference to a bean in the registry as a {{[org.apache.commons.net.ftp.FTPClientConfig|http:<span
class="code-comment">//commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}}
class. Use <span class="code-keyword">this</span> option <span class="code-keyword">if</span>
you need to configure the client according to the FTP Server date format, locale, timezone,
platform etc. See the javadoc {{[FTPClientConfig|http://commons.apache.org/net/apidocs/org/apache/commons/net/ftp/FTPClientConfig.html]}}
<span class="code-keyword">for</span> more documentation.
</span>| {{consumer.recursive}} | {{<span class="code-keyword">true</span>}}/{{<span
class="code-keyword">false</span>}} | If a directory, will look <span class="code-keyword">for</span>
changes in files in all the sub-directories. Is {{<span class="code-keyword">true</span>}}
by <span class="code-keyword">default</span> <span class="code-keyword">for</span>
Camel 1.4 or older. Will change to {{<span class="code-keyword">false</span>}}
by <span class="code-keyword">default</span> value as of Camel 1.5. |
| {{consumer.setNames}} | {{<span class="code-keyword">true</span>}} | *@deprecated*
Used by FTPConsumer. If {{<span class="code-keyword">true</span>}}, Camel will
use the filename the file has on the FTP server. The filename is stored on the In message
in the header, {{FileComponent.HEADER_FILE_NAME}}. *Note:* In Camel 1.4 the <span class="code-keyword">default</span>
value has changed to {{<span class="code-keyword">true</span>}}. |
| {{consumer.delay}} | {{500}} | Delay in milliseconds between each poll. |
| {{consumer.initialDelay}} | {{1000}} | Milliseconds before polling starts. |
| {{consumer.userFixedDelay}} | {{<span class="code-keyword">false</span>}} |
Set to {{<span class="code-keyword">true</span>}} to use fixed delay between polls,
otherwise fixed rate is used. See [ScheduledExecutorService|http:<span class="code-comment">//java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html]
in JDK <span class="code-keyword">for</span> details. |
</span>| {{consumer.regexPattern}} | {{<span class="code-keyword">null</span>}}
| Used by FTPConsumer. Regular expression to use <span class="code-keyword">for</span>
matching files when consuming. |
| {{consumer.exclusiveReadLock}} | {{<span class="code-keyword">false</span>}}|
Camel 1.5: Used by FTPConsumer. If set to {{<span class="code-keyword">true</span>}},
Camel will only poll the ftp files <span class="code-keyword">if</span> it has
exclusive read access to the file (that is, the file is not in the process of being written).
Camel will wait until exclusive access is granted, testing once every second (where the test
consists of Camel attempting to rename the file). If set to {{<span class="code-keyword">false</span>}},
Camel will poll the file even <span class="code-keyword">if</span> it is in the
process of being written. |
| {{consumer.deleteFile}} | {{<span class="code-keyword">false</span>}} | Camel
1.5: Used by FTPConsumer. Flag to set <span class="code-keyword">if</span> the
consumed file should be deleted after it has been downloaded. |
| {{consumer.moveNamePrefix}} | {{<span class="code-keyword">null</span>}} | Camel
1.5: Used by FTPConsumer. The prefix string prepended to the filename when moving it. For
example, to move processed files into the {{done}} directory, set <span class="code-keyword">this</span>
value to {{done/}} |
| {{consumer.moveNamePostfix}} | {{<span class="code-keyword">null</span>}} |
Camel 1.5: Used by FTPConsumer. The postfix string appended to the filename when moving it.
For example to rename processed files from {{foo}} to {{foo.old}}, set <span class="code-keyword">this</span>
value to {{.old}}. |
| {{consumer.excludedNamePrefix}} | {{<span class="code-keyword">null</span>}}
| Camel 1.5: Used by FTPConsumer. Used to exclude files, <span class="code-keyword">if</span>
filename starts with the given prefix. |
| {{consumer.excludedNamePostfix}} | {{<span class="code-keyword">null</span>}}
| Camel 1.5: Used by FTPConsumer. Used to exclude files, <span class="code-keyword">if</span>
filename ends with the given postfix. |
| {{consumer.timestamp}} | {{<span class="code-keyword">false</span>}} | Camel
1.5: *@deprecated* will be removed in Camel 2.0. This option is only <span class="code-keyword">for</span>
backwards compatability. |
| {{expression}} | {{<span class="code-keyword">null</span>}} | Camel 1.5: Use
an expression to dynamically set the filename. This allows you very easily to set dynamic
pattern style filenames. If an expression is set, it takes precedence over the {{org.apache.camel.file.name}}
header. (Note: The header can itself also be an expression.) The {{expression}} option supports
both {{<span class="code-object">String</span>}} and {{Expression}} types. If
the expression is a {{<span class="code-object">String</span>}} type, it is *always*
evaluated using the [File Language]. If the expression is an {{Expression}} type, then the
specified {{Expression}} type is used. This allows you, <span class="code-keyword">for</span>
instance, to use [OGNL] expressions. |
| {{passiveMode}} | {{<span class="code-keyword">false</span>}} | Camel 1.6.0:
Specifies whether to use a passive mode connections. Default is active mode ({{<span class="code-keyword">false</span>}}).
This feature is only <span class="code-keyword">for</span> regular FTP, not SFTP.
|
| {{knownHosts}} | {{<span class="code-keyword">null</span>}} | Camel 1.6.0: Sets
the {{known_hosts}} file, so that the SFTP endpoint can <span class="code-keyword">do</span>
host key verification. |
| {{privateKeyFile}} | {{<span class="code-keyword">null</span>}} | Camel 1.6.0:
Set the <span class="code-keyword">private</span> key file to that the SFTP endpoint
can <span class="code-keyword">do</span> <span class="code-keyword">private</span>
key verification. |
| {{privateKeyFilePassphrase}} | {{<span class="code-keyword">null</span>}} |
Camel 1.6.0: Set the <span class="code-keyword">private</span> key file passphrase
to that the SFTP endpoint can <span class="code-keyword">do</span> <span class="code-keyword">private</span>
key verification. |
{div}

h4. Examples
{{ftp:<span class="code-comment">//someone@someftpserver.com/<span class="code-keyword">public</span>/upload/images/holiday2008?password=secret&amp;binary=<span
class="code-keyword">true</span>}}
</span>{{ftp:<span class="code-comment">//someoneelse@someotherftpserver.co.uk:12049/reports/2008/budget.txt?password=secret&amp;binary=<span
class="code-keyword">false</span>&amp;directory=<span class="code-keyword">false</span>}}
</span>{{ftp:<span class="code-comment">//publicftpserver.com/download}}
</span>
{warning:title=Timestamp}
In Camel 1.4 or older the FTP consumer uses an internal timestamp <span class="code-keyword">for</span>
last polling. This timestamp is used to match <span class="code-keyword">for</span>
<span class="code-keyword">new</span> remote files: <span class="code-keyword">if</span>
remote file modified timestamp &gt; last poll timestamp =&gt; file can be consumed.

In Camel 1.5 <span class="code-keyword">this</span> algorithm has been disabled
by <span class="code-keyword">default</span>, as it is not reliable over the FTP
protocol. FTP Servers only <span class="code-keyword">return</span> file modified
timestamps using {{hh:mm}} (not seconds). And of course the clocks between the client and
server can also be out of sync. Bottom line is that timestamp check <span class="code-keyword">for</span>
FTP protocol should *not* be used. That is why <span class="code-keyword">this</span>
feature is marked as *@deprecated* and will be removed in Camel 2.0.

We encourage you to use a different strategy <span class="code-keyword">for</span>
matching <span class="code-keyword">new</span> remote files: such as deleting
or moving the file after download.
{warning}

{warning:title=FTP does not support concurrency <span class="code-keyword">for</span>
same endpoint}
In Camel 1.x both the FTP consumer and FTP producer (created from the same endpoint) does
not support concurrency (the backing FTP client is not thread safe).
{warning}

{tip:title=More examples}
This component is an extension of the [File] component. So there could be more samples and
details on the [File] component page as well.
{tip}

h3. New <span class="code-keyword">default</span> behavior <span class="code-keyword">for</span>
FTP/SFTP-Consumers in Camel 1.5
The consumer will always skip any file which name starts with a dot, such as {{<span class="code-quote">"."</span>,
<span class="code-quote">".camel"</span>, <span class="code-quote">".m2"</span>
or <span class="code-quote">".groovy"</span>}}. Only files (not directories) is
matched <span class="code-keyword">for</span> valid filename <span class="code-keyword">if</span>
options such as: {{consumer.regexPattern, consumer.excludeNamePrefix, consumer.excludeNamePostfix}}
is used.

The consumer recursive option will be changed from *<span class="code-keyword">true</span>*
to *<span class="code-keyword">false</span>* as the <span class="code-keyword">default</span>
value. We don't feel that Camel out-of-the-box should recursive poll.

The consumer will *not* use timestamp algorithm <span class="code-keyword">for</span>
determine <span class="code-keyword">if</span> a remote file is a <span class="code-keyword">new</span>
file - see warning section above. To use the old behavior of Camel 1.4 or older you can use
the option {{consumer.timestamp=<span class="code-keyword">true</span>}}.

h3. Exclusive Read Lock
The option *readLock* can be used to force Camel *not* to consume files that is currently
in the progress of being written. However <span class="code-keyword">this</span>
option is <span class="code-keyword">default</span> turned off, as it requires
that the user has write access. There are other solutions to avoid consuming files that are
currently being written over FTP, <span class="code-keyword">for</span> instance
you can write the a temporary destination and move the file after it has been written. 

h3. Message Headers

The following message headers can be used to affect the behavior of the component

|| Header || Description ||
| org.apache.camel.file.name | Specifies the output file name (relative to the endpoint directory)
to be used <span class="code-keyword">for</span> the output message when sending
to the endpoint. If <span class="code-keyword">this</span> is not present and
no expression either then a generated message Id is used as filename instead.  |
| org.apache.camel.file.name.produced | New in Camel 1.5: The actual absolute filepath (path
+ name) <span class="code-keyword">for</span> the output file that was written.
This header is set by Camel and its purpose is providing end-users the name of the file that
was written. |
| file.remote.host | The hostname of the remote server |
| file.remote.name | The name of the file consumed from the remote server |
| file.remote.fullName | The fullname of the file consumed from the remote server |

h3. Consumer properties

When using FTPConsumer (downloading files from a FTP Server) the consumer specific properties
from the [File] component should be prefixed with <span class="code-quote">"consumer."</span>.
For example the delay option from File Component should be specified as <span class="code-quote">"consumer.delay=30000"</span>
in the URI. See the samples or some of the unit tests of <span class="code-keyword">this</span>
component.

h3. Filename Expression
In Camel 1.5 we have support <span class="code-keyword">for</span> setting the
filename using an expression. This can be set either using the *expression* option or as a
string based [File Language] expression in the {{org.apache.camel.file.name}} header. See
the [File Language] <span class="code-keyword">for</span> some samples.

h3. Camel 1.x Known issues

See the timestamp warning.

When consuming files (downloading) you must use type conversation to either <span class="code-object">String</span>
or to InputStream <span class="code-keyword">for</span> ASCII and BINARY file
types. 
In Camel 1.4 <span class="code-keyword">this</span> is fixed, as there are build
in type converters <span class="code-keyword">for</span> the ASCII and BINARY
file types, meaning that you <span class="code-keyword">do</span> not need the
convertBodyTo expression.

In Camel 1.4 or below Camel FTPConsumer will poll files regardless <span class="code-keyword">if</span>
the file is currently being written. See the *consumer.exclusiveReadLock* option.

Also in Camel 1.3 since setNames is <span class="code-keyword">default</span>
*<span class="code-keyword">false</span>* then you must explicitly set the filename
using the setHeader expression when consuming from FTP directly to File.
The code below illustrates <span class="code-keyword">this</span>:

</pre>
</div></div>
<p>private String ftpUrl = "ftp://camelrider@localhost:21/public/downloads?password=admin&amp;binary=false";<br/>
private String fileUrl = "file:myfolder/?append=false&amp;noop=true";</p>

<p>return new RouteBuilder() {<br/>
    public void configure() throws Exception <div class="error"><span class="error">Unknown
macro: {
        from(ftpUrl).setHeader(FileComponent.HEADER_FILE_NAME, constant(&quot;downloaded.txt&quot;)).convertBodyTo(String.class).to(fileUrl);
    }</span> </div><br/>
};</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">

Or you can set the option to *<span class="code-keyword">true</span>* as illustrated
below:

</pre>
</div></div>
<p>private String ftpUrl = "ftp://camelrider@localhost:21/public/downloads?password=admin&amp;binary=false&amp;consumer.setNames=true";<br/>
private String fileUrl = "file:myfolder/?append=false&amp;noop=true";</p>

<p>return new RouteBuilder() {<br/>
    public void configure() throws Exception <div class="error"><span class="error">Unknown
macro: {
        from(ftpUrl).convertBodyTo(String.class).to(fileUrl);
    }</span> </div><br/>
};</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">

h3. Sample

In the sample below we setup Camel to download all the reports from the FTP server once every
hour (60 min) as BINARY content and store it as files on the local file system.
{snippet:id=e1|lang=java|url=camel/branches/camel-1.x/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFtpToBinarySampleTest.java}

And the route using Spring DSL:
{code:xml}
  &lt;route&gt;
     &lt;from uri=<span class="code-quote">"ftp:<span class="code-comment">//scott@localhost/<span
class="code-keyword">public</span>/reports?password=tiger&amp;amp;binary=<span
class="code-keyword">true</span>&amp;amp;consumer.delay=60000"</span>/&gt;
</span>     &lt;to uri=<span class="code-quote">"file:<span class="code-comment">//target/test-reports"</span>/&gt;
</span>  &lt;/route&gt;
</pre>
</div></div>

<h4><a name="FTP-Usingexpressionforfilenames"></a>Using expression for filenames</h4>

<p>In this sample we want to move consumed files to a backup folder using today's date
as a sub foldername. Notice that the move happens on the remote FTP server. If you want to
store the downloaded file on your local disk then route it to the <a href="/confluence/display/CAMEL/File"
title="File">File</a> component as the sample above illustrates.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
from(ftpUrl + <span class="code-quote">"&amp;expression=backup/${date:now:yyyyMMdd}/${file:name}"</span>).to(<span
class="code-quote">"..."</span>);
</pre>
</div></div>

<p>See <a href="/confluence/display/CAMEL/File+Language" title="File Language">File
Language</a> for more samples.</p>

<h4><a name="FTP-ConsumingaremoteFTPservertriggeredbyaroute"></a>Consuming
a remote FTP server triggered by a route</h4>
<p>The FTP consumer is built as a scheduled consumer to be used in the <b>from</b>
route. However if you want to start consuming from a FTP server triggered within a route it's
a bit cumbersome to do this in Camel 1.x (we plan to improve this in Camel 2.x). However it's
possible as this code below demonstrates.</p>

<p>In the sample we have a <a href="/confluence/display/CAMEL/SEDA" title="SEDA">SEDA</a>
queue where a message arrives that holds a message containing a filename to poll from a remote
FTP server. So we setup a basic FTP url as:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java"><span class="code-comment">// we use directory=<span
class="code-keyword">false</span> to indicate we only want to consume a single file
</span><span class="code-comment">// we use delay=5000 to use 5 sec delay between
pools to avoid polling a second time before we stop the consumer
</span><span class="code-comment">// <span class="code-keyword">this</span>
is because we only want to run a single poll and get the file
</span><span class="code-comment">// file=getme/ is the path to the folder where
the file is
</span><span class="code-keyword">private</span> <span class="code-object">String</span>
getUrl = <span class="code-quote">"ftp:<span class="code-comment">//admin@localhost:"</span>
+ port + <span class="code-quote">"?password=admin&amp;binary=<span class="code-keyword">false</span>&amp;directory=<span
class="code-keyword">false</span>&amp;consumer.delay=5000&amp;file=getme/"</span>;</span>
</pre>
</div></div>

<p>And then we have the route where we use <a href="/confluence/display/CAMEL/Processor"
title="Processor">Processor</a> within the route so we can use Java code. In this
Java code we create the ftp consumer that downloads the file we want. And after the download
we can get the content of the file and put it in the original exchange that continues being
routed. As this is based on an unit test it routes to a <a href="/confluence/display/CAMEL/Mock"
title="Mock">Mock</a> endpoint.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">from(<span class="code-quote">"seda:start"</span>).process(<span
class="code-keyword">new</span> Processor() {
    <span class="code-keyword">public</span> void process(<span class="code-keyword">final</span>
Exchange exchange) <span class="code-keyword">throws</span> Exception {
        <span class="code-comment">// get the filename from our custome header we want
to get from a remote server
</span>        <span class="code-object">String</span> filename = exchange.getIn().getHeader(<span
class="code-quote">"myfile"</span>, <span class="code-object">String</span>.class);

        <span class="code-comment">// construct the total url <span class="code-keyword">for</span>
the ftp consumer
</span>        <span class="code-object">String</span> url = getUrl + filename;

        <span class="code-comment">// create a ftp endpoint
</span>        Endpoint ftp = context.getEndpoint(url);

        <span class="code-comment">// create a polling consumer so we can poll the remote
ftp file
</span>        PollingConsumer consumer = ftp.createPollingConsumer();
        consumer.start();
        <span class="code-comment">// receive the remote ftp without timeout
</span>        Exchange result = consumer.receive();
        <span class="code-comment">// we must stop the consumer
</span>        consumer.stop();

        <span class="code-comment">// the result is the response from the FTP consumer
(the downloaded file)
</span>        <span class="code-comment">// replace the outher exchange with
the content from the downloaded file
</span>        exchange.getIn().setBody(result.getIn().getBody());
    }
}).to(<span class="code-quote">"mock:result"</span>);
</pre>
</div></div>

<h3><a name="FTP-Debuglogging"></a>Debug logging</h3>
<p>This component has log level <b>TRACE</b> that can be helpful if you
have problems.</p>

<h3><a name="FTP-SeeAlso"></a>See Also</h3>
<ul>
	<li><a href="/confluence/display/CAMEL/Configuring+Camel" title="Configuring Camel">Configuring
Camel</a></li>
	<li><a href="/confluence/display/CAMEL/Component" title="Component">Component</a></li>
	<li><a href="/confluence/display/CAMEL/Endpoint" title="Endpoint">Endpoint</a></li>
	<li><a href="/confluence/display/CAMEL/Getting+Started" title="Getting Started">Getting
Started</a></li>
</ul>

     </div>
     <div id="commentsSection" class="wiki-content pageSection">
       <div style="float: right;">
            <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
       </div>

       <a href="http://cwiki.apache.org/confluence/display/CAMEL/FTP">View Online</a>
       |
       <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=57232&revisedVersion=68&originalVersion=67">View
Change</a>
              |
       <a href="http://cwiki.apache.org/confluence/display/CAMEL/FTP?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message