commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Eckenfels (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (NET-550) Default FTPClient bufferSize results in very slow retrieve transfers
Date Thu, 07 Aug 2014 19:37:14 GMT

    [ https://issues.apache.org/jira/browse/NET-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089664#comment-14089664
] 

Bernd Eckenfels edited comment on NET-550 at 8/7/14 7:36 PM:
-------------------------------------------------------------

Actually looking at 3.4 the buffering situation is a bit better, but still two instances can
be avoided:

{code}
Index: src/main/java/org/apache/commons/net/ftp/FTPClient.java                           
                                    
===================================================================                      
                                    
--- src/main/java/org/apache/commons/net/ftp/FTPClient.java     (Revision 1616569)       
                                    
+++ src/main/java/org/apache/commons/net/ftp/FTPClient.java     (Arbeitskopie)           
                                    
@@ -656,7 +656,7 @@                                                                      
                                    
         if (__fileType == ASCII_FILE_TYPE) {                                            
                                    
             output = new ToNetASCIIOutputStream(getBufferedOutputStream(socket.getOutputStream()));
                         
         } else {                                                                        
                                    
-            output = getBufferedOutputStream(socket.getOutputStream());                 
                                    
+            output = socket.getOutputStream();                                          
                                    
         }                                                                               
                                    
                                                                                         
                                    
         CSL csl = null;                                                                 
                                    
@@ -1878,7 +1878,7 @@                                                                    
                                    
         if (__fileType == ASCII_FILE_TYPE) {                                            
                                    
             input = new FromNetASCIIInputStream(getBufferedInputStream(socket.getInputStream()));
                           
         } else {                                                                        
                                    
-            input = getBufferedInputStream(socket.getInputStream());                    
                                    
+            input = socket.getInputStream();                                            
                                    
         }                                                                               
                                    
                                                                                         
                                    
         CSL csl = null;                                                                 
                                    
{code}

BTW: the default buffer of Util.copyStream is 1k, when removing the above buffered wrappers
I would also increase the default buffer of the Util to saner 8k.


was (Author: b.eckenfels):
Actually looking at 3.4 the buffering situation is a bit better, but still two instances can
be avoided:

{code}
Index: src/main/java/org/apache/commons/net/ftp/FTPClient.java                           
                                    
===================================================================                      
                                    
--- src/main/java/org/apache/commons/net/ftp/FTPClient.java     (Revision 1616569)       
                                    
+++ src/main/java/org/apache/commons/net/ftp/FTPClient.java     (Arbeitskopie)           
                                    
@@ -656,7 +656,7 @@                                                                      
                                    
         if (__fileType == ASCII_FILE_TYPE) {                                            
                                    
             output = new ToNetASCIIOutputStream(getBufferedOutputStream(socket.getOutputStream()));
                         
         } else {                                                                        
                                    
-            output = getBufferedOutputStream(socket.getOutputStream());                 
                                    
+            output = socket.getOutputStream();                                          
                                    
         }                                                                               
                                    
                                                                                         
                                    
         CSL csl = null;                                                                 
                                    
@@ -1878,7 +1878,7 @@                                                                    
                                    
         if (__fileType == ASCII_FILE_TYPE) {                                            
                                    
             input = new FromNetASCIIInputStream(getBufferedInputStream(socket.getInputStream()));
                           
         } else {                                                                        
                                    
-            input = getBufferedInputStream(socket.getInputStream());                    
                                    
+            input = socket.getInputStream();                                            
                                    
         }                                                                               
                                    
                                                                                         
                                    
         CSL csl = null;                                                                 
                                    
{code}

> Default FTPClient bufferSize results in very slow retrieve transfers
> --------------------------------------------------------------------
>
>                 Key: NET-550
>                 URL: https://issues.apache.org/jira/browse/NET-550
>             Project: Commons Net
>          Issue Type: Bug
>    Affects Versions: 3.3
>            Reporter: Geoffrey Hardy
>         Attachments: net-buffercopy.patch
>
>
> While experimenting with FTPClient, I discovered that if I don't call setBufferSize(),
the default value is zero.  This results in retrieveFile() calling the version of InputStream.read()
with no parameters, reading one byte at a time.  For comparison, the downloading a CD ISO
image of about ~648MB took 18m10s with the default settings.  In contrast, calling setBufferSize(8192)
took only 7.9s, an improvement of ~137x.
> Here is some sample code:
> {code:java}
> FTPClient ftp = new FTPClient();
> // ftp.setBufferSize(8192);
> ftp.setControlKeepAliveTimeout(300);
> ftp.setCopyStreamListener(new CopyStreamListener() {
>     @Override
>     public void bytesTransferred(long totalBytesTransferred, int bytesTransferred, long
streamSize) {
>         System.out.println("totalBytesTransferred: " + totalBytesTransferred
>             + ", bytesTransferred: " + bytesTransferred + ", streamSize: " + streamSize);
>     }
>     @Override public void bytesTransferred(CopyStreamEvent event) {}
> });
> ftp.connect(host);
> ftp.login(user, pass);
> ftp.retrieveFile(file, outputStream);
> {code}
> The log message from the stream listener printed lots messages "bytesTransferred: 1"
and totalBytesTransferred incremented by 1 each time.  This corresponds to the part of the
code which reads one byte at a time with {{int inputStream.read()}}.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message