Mailing-List: contact issues-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: issues@commons.apache.org
Date: Wed, 18 Oct 2017 22:46:00 +0000 (UTC)
From: "Gary Gregory (JIRA)" <jira@apache.org>
To: issues@commons.apache.org
Message-ID: <JIRA.13110366.1508349379000.2624.1508366760262@Atlassian.JIRA>
In-Reply-To: <JIRA.13110366.1508349379000@Atlassian.JIRA>
References: <JIRA.13110366.1508349379000@Atlassian.JIRA> <JIRA.13110366.1508349379765@jira-lw-us.apache.org>
Subject: [jira] [Updated] (VFS-647) calling findFiles() causes copyFrom() to
 fail with a partially downloaded file.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 18 Oct 2017 22:46:09 -0000


     [ https://issues.apache.org/jira/browse/VFS-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gary Gregory updated VFS-647:
-----------------------------
    Description: 
When using FileObject.copyFrom(remote, new AllFileSelector()) to download file from remote to local directory. If SftpFileObject.findFiles(new FileDepthSelector(1,1)) is called and finished, then the copyFrom will get interrupted with error pipe closed.

Below are test codes and stack trace error. In real scenario, this all happens within 1 or 2 second time frame. However, with test scenario, I wasn't able to reproduce it easily therefore I had to choose a file that takes around 10 seconds to download and have Thread.sleep(5000) after findFiles() call. Only tested jsch-0.1.52, jsch-0.1.54, commons-vfs2-2.0 and commons-vfs2-2.2.

{code:java}
public class FtpClient {
	
	public static void main(String[] args) {
		if (args.length < 5) {
			throw new RuntimeException("args: host user pass local remote");
		}
		String hostname = args[0];
		String username = args[1];
		String password = args[2];
		int port = 22;
		String local = args[3];
		String remote = args[4];
		final String remoteDir = remote.substring(0, Math.max(0, remote.lastIndexOf("/")));


		Thread t0 = new Thread() {
			public void run() {
				try (Ftp ftp = new Ftp(hostname, port)) {
					ftp.login(username, password);
					ftp.list(remoteDir);
					System.out.println("findFiles() completed.");
				} catch (Exception e) {
					e.printStackTrace();
				}

			}
		};
		
		Thread ti = new Thread() {
			public void run() {
				try (Ftp ftp = new Ftp(hostname, port)) {
					ftp.login(username, password);
					ftp.download(local, remote);
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		};

		t0.start();
		ti.start();	
	}

}


public class Ftp implements AutoCloseable {

	private int timeout = 0;
	private StaticUserAuthenticator userAuth;
	private FileSystemOptions fileSysOpts;
	private FileObject scr = null;
	private FileSystemManager fsm = null;
	private String hostName;
	private int port = 0;


	public Ftp(String remoteHost, int controlPort) throws FileSystemException {
		hostName = remoteHost;
		port = controlPort;
		fsm = VFS.getManager();
	}

	//login into a server with a valid account
	public void login(String user, String password) throws IOException {
		userAuth = new StaticUserAuthenticator(null, user, password); 
		fileSysOpts = new FileSystemOptions();
		DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(fileSysOpts, userAuth);

		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
		SftpFileSystemConfigBuilder.getInstance().setTimeout(fileSysOpts, timeout);
		SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(fileSysOpts, false);
		scr = (FileObject)fsm.resolveFile("sftp://" + hostName, fileSysOpts);
		
		System.out.println("login successfully.");
	}


	public void list(String dirName) throws Exception {
		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + dirName, fileSysOpts);			
		FileObject[] afo = RemoteFo.findFiles(new FileDepthSelector(1,1));		
		Thread.sleep(5000); //key. this must be here (or some other processing that takes time) for error to reproduce consistently.
	}

	public void download(String localPath, String remoteFile) throws FileSystemException {
		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + remoteFile ,fileSysOpts);
		scr = fsm.resolveFile("file:" + localPath);
		scr.copyFrom(RemoteFo, new AllFileSelector());
	}
	
	@Override
	public void close() {
		try{
			FileSystem fs = null;
			if(this.scr!=null){
				this.scr.close();
				fs = this.scr.getFileSystem();
				this.fsm.closeFileSystem(fs);
			}
		}catch(Exception e){
			System.out.println("unable to release ftp connection.");
			e.printStackTrace();
		}
	}


}
{code}

StackTrace:
{noformat}
Oct 18, 2017 12:42:51 PM org.apache.commons.vfs2.VfsLog info
INFO: Using "C:\Users\kenji\AppData\Local\Temp\vfs_cache" as temporary files stor
e.
login successfully.
login successfully.
findFiles() completed.
org.apache.commons.vfs2.FileSystemException: Could not copy "sftp://hostname/path/to/file/FILE1" to "file:///C:/test/ftp/FILE1".
        at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
FileObject.java:1062)
        at Ftp.download(Ftp.java:59)
        at FtpClient$2.run(FtpClient.java:36)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the inpu
t stream for file "sftp://hostname/path/to/file/FILE1".
        at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
tream.close(DefaultFileContent.java:611)
        at org.apache.commons.vfs2.FileUtil.writeContent(FileUtil.java:95)
        at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:114)
        at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
FileObject.java:1053)
        ... 2 more
Caused by: java.io.IOException: Pipe closed
        at java.io.PipedInputStream.read(PipedInputStream.java:307)
        at java.io.PipedInputStream.read(PipedInputStream.java:377)
        at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)
        at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)
        at com.jcraft.jsch.ChannelSftp.access$500(ChannelSftp.java:36)
        at com.jcraft.jsch.ChannelSftp$RequestQueue.cancel(ChannelSftp.java:1238
)
        at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1503)
        at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
        at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
eam.java:129)
        at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
        at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
eam.java:129)
        at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
tream.close(DefaultFileContent.java:607)
        ... 5 more
{noformat}

  was:
When using FileObject.copyFrom(remote, new AllFileSelector()) to download file from remote to local directory. If SftpFileObject.findFiles(new FileDepthSelector(1,1)) is called and finished, then the copyFrom will get interrupted with error pipe closed.

Below are test codes and stack trace error. In real scenario, this all happens within 1 or 2 second time frame. However, with test scenario, I wasn't able to reproduce it easily therefore I had to choose a file that takes around 10 seconds to download and have Thread.sleep(5000) after findFiles() call. Only tested jsch-0.1.52, jsch-0.1.54, commons-vfs2-2.0 and commons-vfs2-2.2.

{code:java}
public class FtpClient {
	
	public static void main(String[] args) {
		if (args.length < 5) {
			throw new RuntimeException("args: host user pass local remote");
		}
		String hostname = args[0];
		String username = args[1];
		String password = args[2];
		int port = 22;
		String local = args[3];
		String remote = args[4];
		final String remoteDir = remote.substring(0, Math.max(0, remote.lastIndexOf("/")));


		Thread t0 = new Thread() {
			public void run() {
				try (Ftp ftp = new Ftp(hostname, port)) {
					ftp.login(username, password);
					ftp.list(remoteDir);
					System.out.println("findFiles() completed.");
				} catch (Exception e) {
					e.printStackTrace();
				}

			}
		};
		
		Thread ti = new Thread() {
			public void run() {
				try (Ftp ftp = new Ftp(hostname, port)) {
					ftp.login(username, password);
					ftp.download(local, remote);
				} catch (Exception e) {
					e.printStackTrace();
				}
			}
		};

		t0.start();
		ti.start();	
	}

}


public class Ftp implements AutoCloseable {

	private int timeout = 0;
	private StaticUserAuthenticator userAuth;
	private FileSystemOptions fileSysOpts;
	private FileObject scr = null;
	private FileSystemManager fsm = null;
	private String hostName;
	private int port = 0;


	public Ftp(String remoteHost, int controlPort) throws FileSystemException {
		hostName = remoteHost;
		port = controlPort;
		fsm = VFS.getManager();
	}

	//login into a server with a valid account
	public void login(String user, String password) throws IOException {
		userAuth = new StaticUserAuthenticator(null, user, password); 
		fileSysOpts = new FileSystemOptions();
		DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(fileSysOpts, userAuth);

		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
		SftpFileSystemConfigBuilder.getInstance().setTimeout(fileSysOpts, timeout);
		SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(fileSysOpts, false);
		scr = (FileObject)fsm.resolveFile("sftp://" + hostName, fileSysOpts);
		
		System.out.println("login successfully.");
	}


	public void list(String dirName) throws Exception {
		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + dirName, fileSysOpts);			
		FileObject[] afo = RemoteFo.findFiles(new FileDepthSelector(1,1));		
		Thread.sleep(5000); //key. this must be here (or some other processing that takes time) for error to reproduce consistently.
	}

	public void download(String localPath, String remoteFile) throws FileSystemException {
		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + remoteFile ,fileSysOpts);
		scr = fsm.resolveFile("file:" + localPath);
		scr.copyFrom(RemoteFo, new AllFileSelector());
	}
	
	@Override
	public void close() {
		try{
			FileSystem fs = null;
			if(this.scr!=null){
				this.scr.close();
				fs = this.scr.getFileSystem();
				this.fsm.closeFileSystem(fs);
			}
		}catch(Exception e){
			System.out.println("unable to release ftp connection.");
			e.printStackTrace();
		}
	}


}
{code}

StackTrace:
Oct 18, 2017 12:42:51 PM org.apache.commons.vfs2.VfsLog info
INFO: Using "C:\Users\kenji\AppData\Local\Temp\vfs_cache" as temporary files stor
e.
login successfully.
login successfully.
findFiles() completed.
org.apache.commons.vfs2.FileSystemException: Could not copy "sftp://hostname/path/to/file/FILE1" to "file:///C:/test/ftp/FILE1".
        at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
FileObject.java:1062)
        at Ftp.download(Ftp.java:59)
        at FtpClient$2.run(FtpClient.java:36)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the inpu
t stream for file "sftp://hostname/path/to/file/FILE1".
        at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
tream.close(DefaultFileContent.java:611)
        at org.apache.commons.vfs2.FileUtil.writeContent(FileUtil.java:95)
        at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:114)
        at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
FileObject.java:1053)
        ... 2 more
Caused by: java.io.IOException: Pipe closed
        at java.io.PipedInputStream.read(PipedInputStream.java:307)
        at java.io.PipedInputStream.read(PipedInputStream.java:377)
        at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)
        at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)
        at com.jcraft.jsch.ChannelSftp.access$500(ChannelSftp.java:36)
        at com.jcraft.jsch.ChannelSftp$RequestQueue.cancel(ChannelSftp.java:1238
)
        at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1503)
        at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
        at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
eam.java:129)
        at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
        at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
eam.java:129)
        at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
tream.close(DefaultFileContent.java:607)
        ... 5 more


> calling findFiles() causes copyFrom() to fail with a partially downloaded file.
> -------------------------------------------------------------------------------
>
>                 Key: VFS-647
>                 URL: https://issues.apache.org/jira/browse/VFS-647
>             Project: Commons VFS
>          Issue Type: Bug
>    Affects Versions: 2.0, 2.2
>         Environment: Windows 7 and Linux Red Hat.
>            Reporter: Kenji
>            Priority: Minor
>
> When using FileObject.copyFrom(remote, new AllFileSelector()) to download file from remote to local directory. If SftpFileObject.findFiles(new FileDepthSelector(1,1)) is called and finished, then the copyFrom will get interrupted with error pipe closed.
> Below are test codes and stack trace error. In real scenario, this all happens within 1 or 2 second time frame. However, with test scenario, I wasn't able to reproduce it easily therefore I had to choose a file that takes around 10 seconds to download and have Thread.sleep(5000) after findFiles() call. Only tested jsch-0.1.52, jsch-0.1.54, commons-vfs2-2.0 and commons-vfs2-2.2.
> {code:java}
> public class FtpClient {
> 	
> 	public static void main(String[] args) {
> 		if (args.length < 5) {
> 			throw new RuntimeException("args: host user pass local remote");
> 		}
> 		String hostname = args[0];
> 		String username = args[1];
> 		String password = args[2];
> 		int port = 22;
> 		String local = args[3];
> 		String remote = args[4];
> 		final String remoteDir = remote.substring(0, Math.max(0, remote.lastIndexOf("/")));
> 		Thread t0 = new Thread() {
> 			public void run() {
> 				try (Ftp ftp = new Ftp(hostname, port)) {
> 					ftp.login(username, password);
> 					ftp.list(remoteDir);
> 					System.out.println("findFiles() completed.");
> 				} catch (Exception e) {
> 					e.printStackTrace();
> 				}
> 			}
> 		};
> 		
> 		Thread ti = new Thread() {
> 			public void run() {
> 				try (Ftp ftp = new Ftp(hostname, port)) {
> 					ftp.login(username, password);
> 					ftp.download(local, remote);
> 				} catch (Exception e) {
> 					e.printStackTrace();
> 				}
> 			}
> 		};
> 		t0.start();
> 		ti.start();	
> 	}
> }
> public class Ftp implements AutoCloseable {
> 	private int timeout = 0;
> 	private StaticUserAuthenticator userAuth;
> 	private FileSystemOptions fileSysOpts;
> 	private FileObject scr = null;
> 	private FileSystemManager fsm = null;
> 	private String hostName;
> 	private int port = 0;
> 	public Ftp(String remoteHost, int controlPort) throws FileSystemException {
> 		hostName = remoteHost;
> 		port = controlPort;
> 		fsm = VFS.getManager();
> 	}
> 	//login into a server with a valid account
> 	public void login(String user, String password) throws IOException {
> 		userAuth = new StaticUserAuthenticator(null, user, password); 
> 		fileSysOpts = new FileSystemOptions();
> 		DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(fileSysOpts, userAuth);
> 		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> 		SftpFileSystemConfigBuilder.getInstance().setTimeout(fileSysOpts, timeout);
> 		SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(fileSysOpts, false);
> 		scr = (FileObject)fsm.resolveFile("sftp://" + hostName, fileSysOpts);
> 		
> 		System.out.println("login successfully.");
> 	}
> 	public void list(String dirName) throws Exception {
> 		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + dirName, fileSysOpts);			
> 		FileObject[] afo = RemoteFo.findFiles(new FileDepthSelector(1,1));		
> 		Thread.sleep(5000); //key. this must be here (or some other processing that takes time) for error to reproduce consistently.
> 	}
> 	public void download(String localPath, String remoteFile) throws FileSystemException {
> 		SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> 		SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + remoteFile ,fileSysOpts);
> 		scr = fsm.resolveFile("file:" + localPath);
> 		scr.copyFrom(RemoteFo, new AllFileSelector());
> 	}
> 	
> 	@Override
> 	public void close() {
> 		try{
> 			FileSystem fs = null;
> 			if(this.scr!=null){
> 				this.scr.close();
> 				fs = this.scr.getFileSystem();
> 				this.fsm.closeFileSystem(fs);
> 			}
> 		}catch(Exception e){
> 			System.out.println("unable to release ftp connection.");
> 			e.printStackTrace();
> 		}
> 	}
> }
> {code}
> StackTrace:
> {noformat}
> Oct 18, 2017 12:42:51 PM org.apache.commons.vfs2.VfsLog info
> INFO: Using "C:\Users\kenji\AppData\Local\Temp\vfs_cache" as temporary files stor
> e.
> login successfully.
> login successfully.
> findFiles() completed.
> org.apache.commons.vfs2.FileSystemException: Could not copy "sftp://hostname/path/to/file/FILE1" to "file:///C:/test/ftp/FILE1".
>         at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1062)
>         at Ftp.download(Ftp.java:59)
>         at FtpClient$2.run(FtpClient.java:36)
> Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the inpu
> t stream for file "sftp://hostname/path/to/file/FILE1".
>         at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:611)
>         at org.apache.commons.vfs2.FileUtil.writeContent(FileUtil.java:95)
>         at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:114)
>         at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1053)
>         ... 2 more
> Caused by: java.io.IOException: Pipe closed
>         at java.io.PipedInputStream.read(PipedInputStream.java:307)
>         at java.io.PipedInputStream.read(PipedInputStream.java:377)
>         at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)
>         at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)
>         at com.jcraft.jsch.ChannelSftp.access$500(ChannelSftp.java:36)
>         at com.jcraft.jsch.ChannelSftp$RequestQueue.cancel(ChannelSftp.java:1238
> )
>         at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1503)
>         at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
>         at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
>         at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
>         at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
>         at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:607)
>         ... 5 more
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)