commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cysneros, Nelson A CIV SPAWARSYSCEN-PACIFIC, 56540" <nelson.cysne...@navy.mil>
Subject RE: FTPClient hangs when downloading large files
Date Mon, 21 Mar 2011 16:24:06 GMT
Thank you everyone for such quick responses.  I wanted to make sure I
follow up with more info when I had it, so here is what I know:

I'm able to download the files using Filezilla. Here is the
command/respond stack: 

Command:	RETR
15MAR11WV010800011MAR15110220-P1BS-052429279010_04_P008.nitf
Response:	150 RETR command started
Response:	221 Goodbye - idle time exceeded
Error:	Connection closed by server
Response:	550 RETR failed...
Error:	File transfer failed after transferring 302.9 MB in 3650 seconds
Status:	Starting download of
/week0308/15MAR11WV010800011MAR15110220-P1BS-052429279010_04_P008.nitf
Command:	PASV
Response:	227 Entering Passive Mode (173,166,178,236,217,144).
Command:	REST 302821425
Response:	350 Restarting at 302821425
Command:	RETR
15MAR11WV010800011MAR15110220-P1BS-052429279010_04_P008.nitf
Response:	150 RETR command started
Response:	226 Transfer Complete
Status:	File transfer successful, transferred 4.9 MB in 58 seconds

It seems that the server is timing out the connection, while the file is
still downloading. Filezilla seems to restart the download where it left
off and continues.

>From the commons-net documentation, it seems that I should get an
FTPConnectionClosedException when this happens.  I then could restart
the download also, but I never get any exception, my application always
hangs using the "retrieveFile" function. 

I then tried using the "retrieveFileStream" function but that also hangs
in the following locations.

If you call the completePendingCommand immediately after the
retrieveFileStream function, then it will hang at the
completePendingCommand function.
------------------------------------------------------------------------
----
InputStream ios = ftpClient.retrieveFileStream(fileName);
boolean commandOK=ftpClient.completePendingCommand();  //Hangs here
                
//write the inputStream to a FileOutputStream
OutputStream out = new FileOutputStream(file);

fos.flush();
fos.close();

int read = 0;
byte[] bytes = new byte[1024];

while ((read = ios.read(bytes)) != -1) {
	out.write(bytes, 0, read);
} 
------------------------------------------------------------------------
--------

If you call the completePendingCommand after the while loop to read the
ios, then it will hang during the while loop.
------------------------------------------------------------------------
----
InputStream ios = ftpClient.retrieveFileStream(fileName);
                
//write the inputStream to a FileOutputStream
OutputStream out = new FileOutputStream(file);

fos.flush();
fos.close();

int read = 0;
byte[] bytes = new byte[1024];

while ((read = ios.read(bytes)) != -1) { //Hangs here
	out.write(bytes, 0, read);
} 

boolean commandOK=ftpClient.completePendingCommand();  
------------------------------------------------------------------------
--------

I probably could get the FTP server admins to bump up the timout time,
but it would be nice to get this to work regardless.

Thanks again for all the advice.





-----Original Message-----
From: Steve Cole [mailto:scole@camsbycbs.com] 
Sent: Wednesday, February 16, 2011 4:59 AM
To: 'Commons Users List'
Subject: RE: FTPClient hangs when downloading large files

Try it with some other FTP client software. If it doesn't work, then the
problem isn't in your java class.


-----Original Message-----
From: Cysneros, Nelson A CIV SPAWARSYSCEN-PACIFIC, 56540
[mailto:nelson.cysneros@navy.mil]
Sent: Tuesday, February 15, 2011 4:50 PM
To: user@commons.apache.org
Subject: FTPClient hangs when downloading large files



I'm using the net-commons v2.2 FTPClient to download files from a server
and it works great for files less then 250MB, but hangs with larger
files.
Wondering if there is anything I can do to get around this issue.  I'm
downloading files up to 500MB in size.  
Thanks in advance.

Here is my source code:
------------------------------------------------------------------------

		FTPClient ftpClient = new FTPClient();
		File file = null;

		//
////////////////////////////////////////////////////////////////////
		// Grab all files from the FTP server and place them in
the specified
		// folder for processing.
	
////////////////////////////////////////////////////////////////////////

			// Log on to FTP server
			try {
				// FTP Server
				ftpClient.enterLocalPassiveMode();
				ftpClient.connect(ftpServer);

				// After connection attempt, you should
check the reply code to verify
				// success.
				int reply = ftpClient.getReplyCode();
				if
(!FTPReply.isPositiveCompletion(reply))
				{
					ftpClient.disconnect();
					logger.error("FTP server refused
connection.");
					System.exit(1);
				}
	
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
				ftpClient.login(m_sFTPCredUN,
m_sFTPCredPW);

			} catch (java.net.ConnectException ex1) {

				logger.info(ex1.getMessage());
				System.out.print("-----FTP Connection
problem: "+ ftpServer+"\n");
				return;
			}catch (Exception ex1) {

				logger.info(ex1.getMessage());
				System.out.print("-----FTP Connection
problem: "+ftpServer+"\n");
				return;
			}
			
			//Download files
			FileOutputStream fos=null;
			try {
				// List the files in the directory
				FTPFile[] files =
ftpClient.listFiles(ftpPath);
				logger.info("Number of files in dir: " +
files.length);
				for (int i = 0; i < files.length; i++) {
					String fileName =
files[i].getName();
				
					// ////////////////
					// Get Files
					// ////////////////
					logger.info("Getting file: " +
fileName);
					file = new
File(m_sInputRapierPath + File.separator
							+
files[i].getName());
					fos = new
FileOutputStream(file);

					ftpClient.retrieveFile(ftpPath +
"/" + files[i].getName(),fos); //Hangs here
					logger.info("File transfer
complete");
					fos.close();

					}

				

			} catch
(org.apache.commons.net.io.CopyStreamException e) {
				logger.info("FTP FETCH PROBLEM(1): " +
e.getMessage());
				//System.out.println("FTP FETCH
PROBLEM(1): " + e.getMessage());
				try {
					if (file != null){
						fos.close();
						file.delete();
					}
				}//try
				catch(IOException ex){
					logger.info("FTP Problem
occured(4) : " + ex.getMessage());
				}//catch

			} catch (Exception ex) {
				logger.info("FTP FETCH PROBLEM(3): " +
ex.getMessage());
				//System.out.println("FTP FETCH
PROBLEM(3): " + e.getMessage());
				try {
					if (file != null){
						fos.close();
						file.delete();
					}
				}//try
				catch(IOException ex1){
					logger.info("FTP Problem
occured(4) : " + ex1.getMessage());
				}//catch
			} finally {
				try {
					if (file != null){
						
						// Logout from the FTP
Server and disconnect
						ftpClient.logout();
	
if(ftpClient.isConnected())
	
ftpClient.disconnect();
					}
				}//try
				catch(IOException ex){
					logger.info("FTP Problem
occured(4) : " + ex.getMessage());
				}//catch
			}//finally

		}// if



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message