camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From davscl...@apache.org
Subject [camel] 01/11: CAMEL-12127: camel-ftp - Add option to turn on logging of transfer activity
Date Mon, 08 Jan 2018 10:20:47 GMT
This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel.git

commit 6653d1cef2d49e5bc90ec82b300cd2e6e2ac2dbb
Author: Claus Ibsen <claus.ibsen@gmail.com>
AuthorDate: Sat Jan 6 15:23:07 2018 +0100

    CAMEL-12127: camel-ftp - Add option to turn on logging of transfer activity
---
 .../camel-ftp/src/main/docs/ftp-component.adoc     |  3 +-
 .../camel-ftp/src/main/docs/ftps-component.adoc    |  3 +-
 .../file/remote/FtpCopyStreamListener.java         | 52 ++++++++++++++++++++++
 .../camel/component/file/remote/FtpEndpoint.java   | 16 +++++++
 .../camel/component/file/remote/FtpOperations.java | 21 ++++++---
 .../component/file/remote/RemoteFileEndpoint.java  |  1 +
 .../remote/FromFileTransferLoggingLevelTest.java   | 48 ++++++++++++++++++++
 7 files changed, 137 insertions(+), 7 deletions(-)

diff --git a/components/camel-ftp/src/main/docs/ftp-component.adoc b/components/camel-ftp/src/main/docs/ftp-component.adoc
index e5f2690..652ec16 100644
--- a/components/camel-ftp/src/main/docs/ftp-component.adoc
+++ b/components/camel-ftp/src/main/docs/ftp-component.adoc
@@ -97,7 +97,7 @@ with the following path and query parameters:
 | *directoryName* | The starting directory |  | String
 |===
 
-==== Query Parameters (104 parameters):
+==== Query Parameters (105 parameters):
 
 [width="100%",cols="2,5,^1,2",options="header"]
 |===
@@ -109,6 +109,7 @@ with the following path and query parameters:
 | *fileName* (common) | Use Expression such as File Language to dynamically set the filename.
For consumers it's used as a filename filter. For producers it's used to evaluate the filename
to write. If an expression is set it take precedence over the CamelFileName header. (Note:
The header itself can also be an Expression). The expression options support both String and
Expression types. If the expression is a String type it is always evaluated using the File
Language. If the expression  [...]
 | *passiveMode* (common) | Sets passive mode connections. Default is active mode connections.
| false | boolean
 | *separator* (common) | Sets the path separator to be used. UNIX = Uses unix style path
separator Windows = Uses windows style path separator Auto = (is default) Use existing path
separator in file name | UNIX | PathSeparator
+| *transferLoggingLevel* (common) | Configure the logging level to use when logging the progress
of upload and download operations. | DEBUG | LoggingLevel
 | *fastExistsCheck* (common) | If set this option to be true camel-ftp will use the list
file directly to check if the file exists. Since some FTP server may not support to list the
file directly if the option is false camel-ftp will use the old way to list the directory
and check if the file exists. This option also influences readLock=changed to control whether
it performs a fast check to update file information or not. This can be used to speed up the
process if the FTP server has a l [...]
 | *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing
Error Handler which mean any exceptions occurred while the consumer is trying to pickup incoming
messages or the likes will now be processed as a message and handled by the routing Error
Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal
with exceptions that will be logged at WARN or ERROR level and ignored. | false | boolean
 | *delete* (consumer) | If true the file will be deleted after it is processed successfully.
| false | boolean
diff --git a/components/camel-ftp/src/main/docs/ftps-component.adoc b/components/camel-ftp/src/main/docs/ftps-component.adoc
index e7dd2cc..8fd9247 100644
--- a/components/camel-ftp/src/main/docs/ftps-component.adoc
+++ b/components/camel-ftp/src/main/docs/ftps-component.adoc
@@ -57,7 +57,7 @@ with the following path and query parameters:
 | *directoryName* | The starting directory |  | String
 |===
 
-==== Query Parameters (112 parameters):
+==== Query Parameters (113 parameters):
 
 [width="100%",cols="2,5,^1,2",options="header"]
 |===
@@ -69,6 +69,7 @@ with the following path and query parameters:
 | *fileName* (common) | Use Expression such as File Language to dynamically set the filename.
For consumers it's used as a filename filter. For producers it's used to evaluate the filename
to write. If an expression is set it take precedence over the CamelFileName header. (Note:
The header itself can also be an Expression). The expression options support both String and
Expression types. If the expression is a String type it is always evaluated using the File
Language. If the expression  [...]
 | *passiveMode* (common) | Sets passive mode connections. Default is active mode connections.
| false | boolean
 | *separator* (common) | Sets the path separator to be used. UNIX = Uses unix style path
separator Windows = Uses windows style path separator Auto = (is default) Use existing path
separator in file name | UNIX | PathSeparator
+| *transferLoggingLevel* (common) | Configure the logging level to use when logging the progress
of upload and download operations. | DEBUG | LoggingLevel
 | *fastExistsCheck* (common) | If set this option to be true camel-ftp will use the list
file directly to check if the file exists. Since some FTP server may not support to list the
file directly if the option is false camel-ftp will use the old way to list the directory
and check if the file exists. This option also influences readLock=changed to control whether
it performs a fast check to update file information or not. This can be used to speed up the
process if the FTP server has a l [...]
 | *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing
Error Handler which mean any exceptions occurred while the consumer is trying to pickup incoming
messages or the likes will now be processed as a message and handled by the routing Error
Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal
with exceptions that will be logged at WARN or ERROR level and ignored. | false | boolean
 | *delete* (consumer) | If true the file will be deleted after it is processed successfully.
| false | boolean
diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java
new file mode 100644
index 0000000..83c242f
--- /dev/null
+++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote;
+
+import org.apache.camel.util.CamelLogger;
+import org.apache.commons.net.io.CopyStreamEvent;
+import org.apache.commons.net.io.CopyStreamListener;
+
+/**
+ * Listener used for logging the progress of the upload or download of files.
+ */
+public class FtpCopyStreamListener implements CopyStreamListener {
+
+    private final CamelLogger logger;
+    private final String fileName;
+    private final boolean download;
+
+    public FtpCopyStreamListener(CamelLogger logger, String fileName, boolean download) {
+        this.logger = logger;
+        this.fileName = fileName;
+        this.download = download;
+    }
+
+    @Override
+    public void bytesTransferred(CopyStreamEvent event) {
+        // not in use
+    }
+
+    @Override
+    public void bytesTransferred(long totalBytesTransferred, int bytesTransferred, long streamSize)
{
+        // stream size is always -1 from the FTP client
+        if (download) {
+            logger.log("Downloading: " + fileName + " (chunk: " + bytesTransferred + ", total
chunk: " + totalBytesTransferred + " bytes)");
+        } else {
+            logger.log("Uploading: " + fileName + " (chunk: " + bytesTransferred + ", total
chunk: " + totalBytesTransferred + " bytes)");
+        }
+    }
+}
diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java
index 846500e..6e20600 100644
--- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java
+++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java
@@ -21,6 +21,7 @@ import java.util.Map;
 
 import org.apache.camel.FailedToCreateConsumerException;
 import org.apache.camel.FailedToCreateProducerException;
+import org.apache.camel.LoggingLevel;
 import org.apache.camel.Processor;
 import org.apache.camel.component.file.GenericFileConfiguration;
 import org.apache.camel.component.file.GenericFileProducer;
@@ -28,6 +29,7 @@ import org.apache.camel.component.file.remote.RemoteFileConfiguration.PathSepara
 import org.apache.camel.spi.ClassResolver;
 import org.apache.camel.spi.UriEndpoint;
 import org.apache.camel.spi.UriParam;
+import org.apache.camel.util.CamelLogger;
 import org.apache.camel.util.PlatformHelper;
 import org.apache.commons.net.ftp.FTPClient;
 import org.apache.commons.net.ftp.FTPClientConfig;
@@ -53,6 +55,8 @@ public class FtpEndpoint<T extends FTPFile> extends RemoteFileEndpoint<FTPFile>
     protected Map<String, Object> ftpClientParameters;
     @UriParam(label = "advanced")
     protected FTPClient ftpClient;
+    @UriParam(label = "common", defaultValue = "DEBUG")
+    protected LoggingLevel transferLoggingLevel = LoggingLevel.DEBUG;
 
     public FtpEndpoint() {
     }
@@ -252,6 +256,18 @@ public class FtpEndpoint<T extends FTPFile> extends RemoteFileEndpoint<FTPFile>
         this.dataTimeout = dataTimeout;
     }
 
+
+    public LoggingLevel getTransferLoggingLevel() {
+        return transferLoggingLevel;
+    }
+
+    /**
+     * Configure the logging level to use when logging the progress of upload and download
operations.
+     */
+    public void setTransferLoggingLevel(LoggingLevel transferLoggingLevel) {
+        this.transferLoggingLevel = transferLoggingLevel;
+    }
+
     @Override
     public char getFileSeparator() {
         // the regular ftp component should use the configured separator
diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
index a6c6d33..9d968d5 100644
--- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
+++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java
@@ -35,6 +35,7 @@ import org.apache.camel.component.file.GenericFile;
 import org.apache.camel.component.file.GenericFileEndpoint;
 import org.apache.camel.component.file.GenericFileExist;
 import org.apache.camel.component.file.GenericFileOperationFailedException;
+import org.apache.camel.util.CamelLogger;
 import org.apache.camel.util.FileUtil;
 import org.apache.camel.util.IOHelper;
 import org.apache.camel.util.ObjectHelper;
@@ -56,7 +57,8 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
     protected final Logger log = LoggerFactory.getLogger(getClass());
     protected final FTPClient client;
     protected final FTPClientConfig clientConfig;
-    protected RemoteFileEndpoint<FTPFile> endpoint;
+    protected final CamelLogger transferLogger = new CamelLogger(LoggerFactory.getLogger(FtpCopyStreamListener.class));
+    protected FtpEndpoint<FTPFile> endpoint;
 
     public FtpOperations(FTPClient client, FTPClientConfig clientConfig) {
         this.client = client;
@@ -64,7 +66,7 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
     }
 
     public void setEndpoint(GenericFileEndpoint<FTPFile> endpoint) {
-        this.endpoint = (RemoteFileEndpoint<FTPFile>) endpoint;
+        this.endpoint = (FtpEndpoint<FTPFile>) endpoint;
     }
 
     public boolean connect(RemoteFileConfiguration configuration) throws GenericFileOperationFailedException
{
@@ -356,8 +358,11 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
                 remoteName = FileUtil.stripPath(name);
             }
 
+            // setup donwload logger for the given file
+            transferLogger.setLevel(endpoint.getTransferLoggingLevel());
+            client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, remoteName,
true));
+
             log.trace("Client retrieveFile: {}", remoteName);
-            
             if (endpoint.getConfiguration().isStreamDownload()) {
                 InputStream is = client.retrieveFileStream(remoteName); 
                 target.setBody(is);
@@ -455,6 +460,10 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
                 remoteName = FileUtil.stripPath(name);
             }
 
+            // setup donwload logger for the given file
+            transferLogger.setLevel(endpoint.getTransferLoggingLevel());
+            client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, remoteName,
true));
+
             log.trace("Client retrieveFile: {}", remoteName);
             result = client.retrieveFile(remoteName, os);
 
@@ -580,6 +589,9 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
 
             final StopWatch watch = new StopWatch();
             boolean answer;
+            // setup upload logger for the given file
+            transferLogger.setLevel(endpoint.getTransferLoggingLevel());
+            client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, targetName,
false));
             log.debug("About to store file: {} using stream: {}", targetName, is);
             if (endpoint.getFileExist() == GenericFileExist.Append) {
                 log.trace("Client appendFile: {}", targetName);
@@ -599,7 +611,7 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
             exchange.getIn().setHeader(FtpConstants.FTP_REPLY_STRING, client.getReplyString());
 
             // after storing file, we may set chmod on the file
-            String chmod = ((FtpConfiguration) endpoint.getConfiguration()).getChmod();
+            String chmod = endpoint.getConfiguration().getChmod();
             if (ObjectHelper.isNotEmpty(chmod)) {
                 log.debug("Setting chmod: {} on file: {}", chmod, targetName);
                 String command = "chmod " + chmod + " " + targetName;
@@ -607,7 +619,6 @@ public class FtpOperations implements RemoteFileOperations<FTPFile>
{
                 boolean success = client.sendSiteCommand(command);
                 log.trace("Client sendSiteCommand successful: {}", success);
             }
-            
 
             return answer;
 
diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java
b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java
index a037afd..d59ba41 100644
--- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java
+++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java
@@ -19,6 +19,7 @@ package org.apache.camel.component.file.remote;
 import java.util.Map;
 
 import org.apache.camel.Exchange;
+import org.apache.camel.LoggingLevel;
 import org.apache.camel.PollingConsumer;
 import org.apache.camel.Processor;
 import org.apache.camel.component.file.GenericFile;
diff --git a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java
b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java
new file mode 100644
index 0000000..f743bef
--- /dev/null
+++ b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.file.remote;
+
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.junit.Test;
+
+/**
+ * @version 
+ */
+public class FromFileTransferLoggingLevelTest extends FtpServerTestSupport {
+
+    protected String getFtpUrl() {
+        return "ftp://admin@localhost:" + getPort() + "/tmp2/camel?password=admin&transferLoggingLevel=INFO";
+    }
+
+    @Test
+    public void testTransferLoggingLevel() throws Exception {
+        MockEndpoint mock = getMockEndpoint("mock:result");
+        mock.expectedMessageCount(2);
+
+        assertMockEndpointsSatisfied();
+    }
+
+    protected RouteBuilder createRouteBuilder() throws Exception {
+        return new RouteBuilder() {
+            public void configure() throws Exception {
+                from(getFtpUrl()).to("mock:result");
+                from("file:src/main/data?noop=true").to(getFtpUrl());
+            }
+        };
+    }
+}
\ No newline at end of file

-- 
To stop receiving notification emails like this one, please contact
"commits@camel.apache.org" <commits@camel.apache.org>.

Mime
View raw message