commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <seb...@gmail.com>
Subject Re: [net] Problem with completePendingCommand and large files > 600MB
Date Wed, 25 Oct 2017 23:05:15 GMT
Note: curl supports an FTP range parameter.
It seems it does this by sending ABOR after the required data has arrived.

However I just tried and it does not work with the server I used.

You could try using it with your server and see if it works:

$ curl -v -r nnn-mmm -o file URL

If so you may be able to use ABOR in your app for that server.


On 25 October 2017 at 23:57, sebb <sebbaz@gmail.com> wrote:
> What you are doing is stopping reading from the server before the file
> has been fully sent.
> Unless the server is expecting that, it may complain (the fact that it
> works for smaller files may be because the server has already sent the
> data).
>
> AFAICT there is no standard way to tell the server to only send a
> portion of the file.
>
> A work-round would be to fetch all the data but stop writing it to
> disk when you have got what you want.
>
> Another way might be to cancel the transfer after enough data has been received.
> However that relies on the server being able to process control
> channel requests whilst sending data.
>
> Or just live with the error message.
>
> It's not a bug in NET.
>
>
> On 25 October 2017 at 23:30, Bernd Eckenfels <ecki@zusammenkunft.net> wrote:
>> 426 sounds like a server error, did you try to use a different client with your server?
>>
>> Gruss
>> Bernd
>> --
>> http://bernd.eckenfels.net
>> ________________________________
>> From: SeungWook Kim <seungwook.kim@weather.com>
>> Sent: Wednesday, October 25, 2017 6:34:02 PM
>> To: user@commons.apache.org
>> Subject: [net] Problem with completePendingCommand and large files > 600MB
>>
>> Hi,
>>   I am having a problem with commons-net version* 3.6* and with large files
>>> 600MB and offset and length that does not include the whole file.
>>
>> I have included the test code below that appears to fail with large files >
>> 600 MB and an offset and length that does not include the whole file. If I
>> choose a smaller file like 90KB with offset and length, it works
>> successfully. If I choose a large file > 600MB with offset and length that
>> INCLUDES the entire file, it also works successfully. The
>> completePendingCommand replies back with "426 Failure writing network
>> stream" when the file is large > 600MB AND the offset and length is not the
>> entire file.
>>
>>   I believe this is a bug but not sure. I did a quick look in the postings
>> and did not see anything that is exactly like it. Any feedback is greatly
>> appreciated.
>>
>>
>>
>> *** Need to specify serverName, userName,...absOutputFile below***
>> ***Start of Test code***
>>
>> public class FtpTest {
>>     private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>>
>>     private static final int DEFAULT_TIMEOUT = 10*1000;
>>
>>     @Before
>>     public void setup() {
>>
>>     }
>>
>>     @Test
>>     public void test() {
>>         String serverName = "***";
>>         String userName = "***";
>>         String password = "***";
>>         String absRemoteFileName = "***";
>>         final long offset = ***;
>>         final long length = ***;
>>         String absOutputFile = "***";
>>
>>         FTPClient ftpClient = new FTPClient();
>>         try {
>>             ftpClient.connect(serverName);
>>             int reply = ftpClient.getReplyCode();
>>             if (!FTPReply.isPositiveCompletion(reply)) {
>>                 String lastReply = ftpClient.getReplyString();
>>                 fail(lastReply);
>>             }
>>             ftpClient.addProtocolCommandListener(new
>> PrintCommandListener(new PrintWriter(System.out),
>>                     true));
>>             ftpClient.setKeepAlive(true);
>>             ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
>>             //turn on SO_LINGER w/ timeout of 0 to cause connection to be
>> aborted immediately and to discard any
>>             // pending data.
>>             ftpClient.setSoLinger(true, 0);
>>             if(ftpClient.login(userName, password)) {
>>                 ftpClient.enterLocalPassiveMode();
>>                 if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
>> {
>>                     log.info("Successfully connected in Passive Mode and
>> logged into {}", serverName);
>>
>>                     ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
>>                     ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
>>                     ftpClient.setRestartOffset(offset);
>>                     File writeToFile = new File(absOutputFile);
>>
>>                     try (InputStream inputStream =
>> ftpClient.retrieveFileStream(absRemoteFileName);
>>                          OutputStream outputStream =
>> FileUtils.openOutputStream(writeToFile)) {
>>                         if(inputStream == null || outputStream == null) {
>>                             fail("One of the input/output stream is null");
>>                         }
>>                         final int BUFFER_SIZE = 4096;
>>                         byte[] buffer = new byte[BUFFER_SIZE];
>>                         int bytesRead;
>>                         int len = (int)length;
>>                         int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
>> : len;
>>                         while ((bytesRead = inputStream.read(buffer, 0,
>> bufferSize)) != -1) {
>>                             len -= bytesRead;
>>                             bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
>> len;
>>
>>                             outputStream.write(buffer, 0, bytesRead);
>>
>>                             if(len <= 0) {
>>                                 break;
>>                             }
>>                         }
>>
>>
>>                     } finally {
>>                         if(ftpClient.completePendingCommand()) {
>>                             log.info("Successful complete pending command");
>>                         } else {
>>                             fail("Unsuccessful complete pending command");
>>                         }
>>                     }
>>
>>                 } else {
>>                     fail("Problem entering passive mode. All modern ftp
>> server supports passive mode, but " +
>>                             serverName + " doesn't??");
>>
>>                 }
>>             } else {
>>                 fail("Bad credentials to server " + serverName);
>>             }
>>
>>         } catch (Exception e) {
>>             fail(e.toString() + "\n" + e.getStackTrace().toString());
>>         }
>>     }
>> }
>>
>> ***End of Test code***
>>
>> Thank you
>> -Seung

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message