commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Eckenfels (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (VFS-647) calling findFiles() causes copyFrom() to fail with a partially downloaded file.
Date Sat, 04 Nov 2017 01:57:00 GMT

    [ https://issues.apache.org/jira/browse/VFS-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238470#comment-16238470
] 

Bernd Eckenfels edited comment on VFS-647 at 11/4/17 1:56 AM:
--------------------------------------------------------------

It looks like your two threads both close the filesystems and the fsm concurrently. You need
to wait for all filesystems operations to be finished. Fsm is a singleton and getFilesystem()
is also shared if it has the same parameters on resolve.
There is even a warning in the 2.2 Javadoc of getManager about this:

https://commons.apache.org/proper/commons-vfs/apidocs/org/apache/commons/vfs2/VFS.html#getManager--


was (Author: b.eckenfels):
It looks like your two threads both close the filesystems and the fsm concurrently. You need
to wait for all filesystems operations to be finished. Fsm is a singleton and getFilesystem()
is also shared if it has the same parameters on resolve.

> calling findFiles() causes copyFrom() to fail with a partially downloaded file.
> -------------------------------------------------------------------------------
>
>                 Key: VFS-647
>                 URL: https://issues.apache.org/jira/browse/VFS-647
>             Project: Commons VFS
>          Issue Type: Bug
>    Affects Versions: 2.0, 2.2
>         Environment: Windows 7 and Linux Red Hat.
>            Reporter: Kenji
>            Priority: Minor
>
> Edit: The problem isn't specific to findFiles. All I need to do is have 1 thread downloads
the file, and the other thread just log in, have a while loop for x seconds where x < download
time (using Thread.sleep(x) causes interruption to happen immediately). Once x seconds passed,
the download will get interupted with pipe closed. Using the same hostname but different username
and password does not cause interruption.
> Original description:
> When using FileObject.copyFrom(remote, new AllFileSelector()) to download file from remote
to local directory. If SftpFileObject.findFiles(new FileDepthSelector(1,1)) is called and
finished, then the copyFrom will get interrupted with error pipe closed.
> Below are test codes and stack trace error. In real scenario, this all happens within
1 or 2 second time frame. However, with test scenario, I wasn't able to reproduce it easily
therefore I had to choose a file that takes around 10 seconds to download and have Thread.sleep(5000)
after findFiles() call. Only tested jsch-0.1.52, jsch-0.1.54, commons-vfs2-2.0 and commons-vfs2-2.2.
> {code:java}
> public class FtpClient {
> 	
> 	public static void main(String[] args) {
> 		if (args.length < 5) {
> 			throw new RuntimeException("args: host user pass local remote");
> 		}
> 		String hostname = args[0];
> 		String username = args[1];
> 		String password = args[2];
> 		int port = 22;
> 		String local = args[3];
> 		String remote = args[4];
> 		final String remoteDir = remote.substring(0, Math.max(0, remote.lastIndexOf("/")));
> 		Thread t0 = new Thread() {
> 			public void run() {
> 				try (Ftp ftp = new Ftp(hostname, port)) {
> 					ftp.login(username, password);
> 					ftp.list(remoteDir);
> 					System.out.println("findFiles() completed.");
> 				} catch (Exception e) {
> 					e.printStackTrace();
> 				}
> 			}
> 		};
> 		
> 		Thread ti = new Thread() {
> 			public void run() {
> 				try (Ftp ftp = new Ftp(hostname, port)) {
> 					ftp.login(username, password);
> 					ftp.download(local, remote);
> 				} catch (Exception e) {
> 					e.printStackTrace();
> 				}
> 			}
> 		};
> 		t0.start();
> 		ti.start();	
> 	}
> }
> public class Ftp implements AutoCloseable {
> 	private int timeout = 0;
> 	private StaticUserAuthenticator userAuth;
> 	private FileSystemOptions fileSysOpts;
> 	private FileObject scr = null;
> 	private FileSystemManager fsm = null;
> 	private String hostName;
> 	private int port = 0;
> 	public Ftp(String remoteHost, int controlPort) throws FileSystemException {
> 		hostName = remoteHost;
> 		port = controlPort;
> 		fsm = VFS.getManager();
> 	}
> 	//login into a server with a valid account
> 	public void login(String user, String password) throws IOException {
> 		userAuth = new StaticUserAuthenticator(null, user, password); 
> 		fileSysOpts = new FileSystemOptions();
> 		DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(fileSysOpts, userAuth);
> 		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> 		SftpFileSystemConfigBuilder.getInstance().setTimeout(fileSysOpts, timeout);
> 		SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(fileSysOpts, false);
> 		scr = (FileObject)fsm.resolveFile("sftp://" + hostName, fileSysOpts);
> 		
> 		System.out.println("login successfully.");
> 	}
> 	public void list(String dirName) throws Exception {
> 		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + dirName,
fileSysOpts);			
> 		FileObject[] afo = RemoteFo.findFiles(new FileDepthSelector(1,1));		
> 		Thread.sleep(5000); //key. this must be here (or some other processing that takes time)
for error to reproduce consistently.
> 	}
> 	public void download(String localPath, String remoteFile) throws FileSystemException
{
> 		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> 		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + remoteFile
,fileSysOpts);
> 		scr = fsm.resolveFile("file:" + localPath);
> 		scr.copyFrom(RemoteFo, new AllFileSelector());
> 	}
> 	
> 	@Override
> 	public void close() {
> 		try{
> 			FileSystem fs = null;
> 			if(this.scr!=null){
> 				this.scr.close();
> 				fs = this.scr.getFileSystem();
> 				this.fsm.closeFileSystem(fs);
> 			}
> 		}catch(Exception e){
> 			System.out.println("unable to release ftp connection.");
> 			e.printStackTrace();
> 		}
> 	}
> }
> {code}
> StackTrace:
> {noformat}
> Oct 18, 2017 12:42:51 PM org.apache.commons.vfs2.VfsLog info
> INFO: Using "C:\Users\kenji\AppData\Local\Temp\vfs_cache" as temporary files stor
> e.
> login successfully.
> login successfully.
> findFiles() completed.
> org.apache.commons.vfs2.FileSystemException: Could not copy "sftp://hostname/path/to/file/FILE1"
to "file:///C:/test/ftp/FILE1".
>         at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1062)
>         at Ftp.download(Ftp.java:59)
>         at FtpClient$2.run(FtpClient.java:36)
> Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the inpu
> t stream for file "sftp://hostname/path/to/file/FILE1".
>         at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:611)
>         at org.apache.commons.vfs2.FileUtil.writeContent(FileUtil.java:95)
>         at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:114)
>         at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1053)
>         ... 2 more
> Caused by: java.io.IOException: Pipe closed
>         at java.io.PipedInputStream.read(PipedInputStream.java:307)
>         at java.io.PipedInputStream.read(PipedInputStream.java:377)
>         at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)
>         at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)
>         at com.jcraft.jsch.ChannelSftp.access$500(ChannelSftp.java:36)
>         at com.jcraft.jsch.ChannelSftp$RequestQueue.cancel(ChannelSftp.java:1238
> )
>         at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1503)
>         at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
>         at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
>         at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
>         at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
>         at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:607)
>         ... 5 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message