From commits-return-60678-archive-asf-public=cust-asf.ponee.io@camel.apache.org Mon Jan 8 11:20:54 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 9E80E180787 for ; Mon, 8 Jan 2018 11:20:54 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 8E59C160C2C; Mon, 8 Jan 2018 10:20:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 665F7160C3E for ; Mon, 8 Jan 2018 11:20:53 +0100 (CET) Received: (qmail 84228 invoked by uid 500); 8 Jan 2018 10:20:52 -0000 Mailing-List: contact commits-help@camel.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@camel.apache.org Delivered-To: mailing list commits@camel.apache.org Received: (qmail 84120 invoked by uid 99); 8 Jan 2018 10:20:52 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Jan 2018 10:20:52 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 6D5B782095; Mon, 8 Jan 2018 10:20:49 +0000 (UTC) Date: Mon, 08 Jan 2018 10:20:47 +0000 To: "commits@camel.apache.org" Subject: [camel] 01/11: CAMEL-12127: camel-ftp - Add option to turn on logging of transfer activity MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit From: davsclaus@apache.org In-Reply-To: <151540684682.10346.7742608242861948582@gitbox.apache.org> References: <151540684682.10346.7742608242861948582@gitbox.apache.org> X-Git-Host: gitbox.apache.org X-Git-Repo: camel X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Rev: 6653d1cef2d49e5bc90ec82b300cd2e6e2ac2dbb X-Git-NotificationType: diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated Message-Id: <20180108102049.6D5B782095@gitbox.apache.org> This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/camel.git commit 6653d1cef2d49e5bc90ec82b300cd2e6e2ac2dbb Author: Claus Ibsen AuthorDate: Sat Jan 6 15:23:07 2018 +0100 CAMEL-12127: camel-ftp - Add option to turn on logging of transfer activity --- .../camel-ftp/src/main/docs/ftp-component.adoc | 3 +- .../camel-ftp/src/main/docs/ftps-component.adoc | 3 +- .../file/remote/FtpCopyStreamListener.java | 52 ++++++++++++++++++++++ .../camel/component/file/remote/FtpEndpoint.java | 16 +++++++ .../camel/component/file/remote/FtpOperations.java | 21 ++++++--- .../component/file/remote/RemoteFileEndpoint.java | 1 + .../remote/FromFileTransferLoggingLevelTest.java | 48 ++++++++++++++++++++ 7 files changed, 137 insertions(+), 7 deletions(-) diff --git a/components/camel-ftp/src/main/docs/ftp-component.adoc b/components/camel-ftp/src/main/docs/ftp-component.adoc index e5f2690..652ec16 100644 --- a/components/camel-ftp/src/main/docs/ftp-component.adoc +++ b/components/camel-ftp/src/main/docs/ftp-component.adoc @@ -97,7 +97,7 @@ with the following path and query parameters: | *directoryName* | The starting directory | | String |=== -==== Query Parameters (104 parameters): +==== Query Parameters (105 parameters): [width="100%",cols="2,5,^1,2",options="header"] |=== @@ -109,6 +109,7 @@ with the following path and query parameters: | *fileName* (common) | Use Expression such as File Language to dynamically set the filename. For consumers it's used as a filename filter. For producers it's used to evaluate the filename to write. If an expression is set it take precedence over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type it is always evaluated using the File Language. If the expression [...] | *passiveMode* (common) | Sets passive mode connections. Default is active mode connections. | false | boolean | *separator* (common) | Sets the path separator to be used. UNIX = Uses unix style path separator Windows = Uses windows style path separator Auto = (is default) Use existing path separator in file name | UNIX | PathSeparator +| *transferLoggingLevel* (common) | Configure the logging level to use when logging the progress of upload and download operations. | DEBUG | LoggingLevel | *fastExistsCheck* (common) | If set this option to be true camel-ftp will use the list file directly to check if the file exists. Since some FTP server may not support to list the file directly if the option is false camel-ftp will use the old way to list the directory and check if the file exists. This option also influences readLock=changed to control whether it performs a fast check to update file information or not. This can be used to speed up the process if the FTP server has a l [...] | *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing Error Handler which mean any exceptions occurred while the consumer is trying to pickup incoming messages or the likes will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions that will be logged at WARN or ERROR level and ignored. | false | boolean | *delete* (consumer) | If true the file will be deleted after it is processed successfully. | false | boolean diff --git a/components/camel-ftp/src/main/docs/ftps-component.adoc b/components/camel-ftp/src/main/docs/ftps-component.adoc index e7dd2cc..8fd9247 100644 --- a/components/camel-ftp/src/main/docs/ftps-component.adoc +++ b/components/camel-ftp/src/main/docs/ftps-component.adoc @@ -57,7 +57,7 @@ with the following path and query parameters: | *directoryName* | The starting directory | | String |=== -==== Query Parameters (112 parameters): +==== Query Parameters (113 parameters): [width="100%",cols="2,5,^1,2",options="header"] |=== @@ -69,6 +69,7 @@ with the following path and query parameters: | *fileName* (common) | Use Expression such as File Language to dynamically set the filename. For consumers it's used as a filename filter. For producers it's used to evaluate the filename to write. If an expression is set it take precedence over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type it is always evaluated using the File Language. If the expression [...] | *passiveMode* (common) | Sets passive mode connections. Default is active mode connections. | false | boolean | *separator* (common) | Sets the path separator to be used. UNIX = Uses unix style path separator Windows = Uses windows style path separator Auto = (is default) Use existing path separator in file name | UNIX | PathSeparator +| *transferLoggingLevel* (common) | Configure the logging level to use when logging the progress of upload and download operations. | DEBUG | LoggingLevel | *fastExistsCheck* (common) | If set this option to be true camel-ftp will use the list file directly to check if the file exists. Since some FTP server may not support to list the file directly if the option is false camel-ftp will use the old way to list the directory and check if the file exists. This option also influences readLock=changed to control whether it performs a fast check to update file information or not. This can be used to speed up the process if the FTP server has a l [...] | *bridgeErrorHandler* (consumer) | Allows for bridging the consumer to the Camel routing Error Handler which mean any exceptions occurred while the consumer is trying to pickup incoming messages or the likes will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions that will be logged at WARN or ERROR level and ignored. | false | boolean | *delete* (consumer) | If true the file will be deleted after it is processed successfully. | false | boolean diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java new file mode 100644 index 0000000..83c242f --- /dev/null +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpCopyStreamListener.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.component.file.remote; + +import org.apache.camel.util.CamelLogger; +import org.apache.commons.net.io.CopyStreamEvent; +import org.apache.commons.net.io.CopyStreamListener; + +/** + * Listener used for logging the progress of the upload or download of files. + */ +public class FtpCopyStreamListener implements CopyStreamListener { + + private final CamelLogger logger; + private final String fileName; + private final boolean download; + + public FtpCopyStreamListener(CamelLogger logger, String fileName, boolean download) { + this.logger = logger; + this.fileName = fileName; + this.download = download; + } + + @Override + public void bytesTransferred(CopyStreamEvent event) { + // not in use + } + + @Override + public void bytesTransferred(long totalBytesTransferred, int bytesTransferred, long streamSize) { + // stream size is always -1 from the FTP client + if (download) { + logger.log("Downloading: " + fileName + " (chunk: " + bytesTransferred + ", total chunk: " + totalBytesTransferred + " bytes)"); + } else { + logger.log("Uploading: " + fileName + " (chunk: " + bytesTransferred + ", total chunk: " + totalBytesTransferred + " bytes)"); + } + } +} diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java index 846500e..6e20600 100644 --- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpEndpoint.java @@ -21,6 +21,7 @@ import java.util.Map; import org.apache.camel.FailedToCreateConsumerException; import org.apache.camel.FailedToCreateProducerException; +import org.apache.camel.LoggingLevel; import org.apache.camel.Processor; import org.apache.camel.component.file.GenericFileConfiguration; import org.apache.camel.component.file.GenericFileProducer; @@ -28,6 +29,7 @@ import org.apache.camel.component.file.remote.RemoteFileConfiguration.PathSepara import org.apache.camel.spi.ClassResolver; import org.apache.camel.spi.UriEndpoint; import org.apache.camel.spi.UriParam; +import org.apache.camel.util.CamelLogger; import org.apache.camel.util.PlatformHelper; import org.apache.commons.net.ftp.FTPClient; import org.apache.commons.net.ftp.FTPClientConfig; @@ -53,6 +55,8 @@ public class FtpEndpoint extends RemoteFileEndpoint protected Map ftpClientParameters; @UriParam(label = "advanced") protected FTPClient ftpClient; + @UriParam(label = "common", defaultValue = "DEBUG") + protected LoggingLevel transferLoggingLevel = LoggingLevel.DEBUG; public FtpEndpoint() { } @@ -252,6 +256,18 @@ public class FtpEndpoint extends RemoteFileEndpoint this.dataTimeout = dataTimeout; } + + public LoggingLevel getTransferLoggingLevel() { + return transferLoggingLevel; + } + + /** + * Configure the logging level to use when logging the progress of upload and download operations. + */ + public void setTransferLoggingLevel(LoggingLevel transferLoggingLevel) { + this.transferLoggingLevel = transferLoggingLevel; + } + @Override public char getFileSeparator() { // the regular ftp component should use the configured separator diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java index a6c6d33..9d968d5 100644 --- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/FtpOperations.java @@ -35,6 +35,7 @@ import org.apache.camel.component.file.GenericFile; import org.apache.camel.component.file.GenericFileEndpoint; import org.apache.camel.component.file.GenericFileExist; import org.apache.camel.component.file.GenericFileOperationFailedException; +import org.apache.camel.util.CamelLogger; import org.apache.camel.util.FileUtil; import org.apache.camel.util.IOHelper; import org.apache.camel.util.ObjectHelper; @@ -56,7 +57,8 @@ public class FtpOperations implements RemoteFileOperations { protected final Logger log = LoggerFactory.getLogger(getClass()); protected final FTPClient client; protected final FTPClientConfig clientConfig; - protected RemoteFileEndpoint endpoint; + protected final CamelLogger transferLogger = new CamelLogger(LoggerFactory.getLogger(FtpCopyStreamListener.class)); + protected FtpEndpoint endpoint; public FtpOperations(FTPClient client, FTPClientConfig clientConfig) { this.client = client; @@ -64,7 +66,7 @@ public class FtpOperations implements RemoteFileOperations { } public void setEndpoint(GenericFileEndpoint endpoint) { - this.endpoint = (RemoteFileEndpoint) endpoint; + this.endpoint = (FtpEndpoint) endpoint; } public boolean connect(RemoteFileConfiguration configuration) throws GenericFileOperationFailedException { @@ -356,8 +358,11 @@ public class FtpOperations implements RemoteFileOperations { remoteName = FileUtil.stripPath(name); } + // setup donwload logger for the given file + transferLogger.setLevel(endpoint.getTransferLoggingLevel()); + client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, remoteName, true)); + log.trace("Client retrieveFile: {}", remoteName); - if (endpoint.getConfiguration().isStreamDownload()) { InputStream is = client.retrieveFileStream(remoteName); target.setBody(is); @@ -455,6 +460,10 @@ public class FtpOperations implements RemoteFileOperations { remoteName = FileUtil.stripPath(name); } + // setup donwload logger for the given file + transferLogger.setLevel(endpoint.getTransferLoggingLevel()); + client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, remoteName, true)); + log.trace("Client retrieveFile: {}", remoteName); result = client.retrieveFile(remoteName, os); @@ -580,6 +589,9 @@ public class FtpOperations implements RemoteFileOperations { final StopWatch watch = new StopWatch(); boolean answer; + // setup upload logger for the given file + transferLogger.setLevel(endpoint.getTransferLoggingLevel()); + client.setCopyStreamListener(new FtpCopyStreamListener(transferLogger, targetName, false)); log.debug("About to store file: {} using stream: {}", targetName, is); if (endpoint.getFileExist() == GenericFileExist.Append) { log.trace("Client appendFile: {}", targetName); @@ -599,7 +611,7 @@ public class FtpOperations implements RemoteFileOperations { exchange.getIn().setHeader(FtpConstants.FTP_REPLY_STRING, client.getReplyString()); // after storing file, we may set chmod on the file - String chmod = ((FtpConfiguration) endpoint.getConfiguration()).getChmod(); + String chmod = endpoint.getConfiguration().getChmod(); if (ObjectHelper.isNotEmpty(chmod)) { log.debug("Setting chmod: {} on file: {}", chmod, targetName); String command = "chmod " + chmod + " " + targetName; @@ -607,7 +619,6 @@ public class FtpOperations implements RemoteFileOperations { boolean success = client.sendSiteCommand(command); log.trace("Client sendSiteCommand successful: {}", success); } - return answer; diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java index a037afd..d59ba41 100644 --- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java @@ -19,6 +19,7 @@ package org.apache.camel.component.file.remote; import java.util.Map; import org.apache.camel.Exchange; +import org.apache.camel.LoggingLevel; import org.apache.camel.PollingConsumer; import org.apache.camel.Processor; import org.apache.camel.component.file.GenericFile; diff --git a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java new file mode 100644 index 0000000..f743bef --- /dev/null +++ b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileTransferLoggingLevelTest.java @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.component.file.remote; + +import org.apache.camel.builder.RouteBuilder; +import org.apache.camel.component.mock.MockEndpoint; +import org.junit.Test; + +/** + * @version + */ +public class FromFileTransferLoggingLevelTest extends FtpServerTestSupport { + + protected String getFtpUrl() { + return "ftp://admin@localhost:" + getPort() + "/tmp2/camel?password=admin&transferLoggingLevel=INFO"; + } + + @Test + public void testTransferLoggingLevel() throws Exception { + MockEndpoint mock = getMockEndpoint("mock:result"); + mock.expectedMessageCount(2); + + assertMockEndpointsSatisfied(); + } + + protected RouteBuilder createRouteBuilder() throws Exception { + return new RouteBuilder() { + public void configure() throws Exception { + from(getFtpUrl()).to("mock:result"); + from("file:src/main/data?noop=true").to(getFtpUrl()); + } + }; + } +} \ No newline at end of file -- To stop receiving notification emails like this one, please contact "commits@camel.apache.org" .