commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Vercammen <nick.vercam...@intec.ugent.be>
Subject [VFS] Uploading large files with sftp
Date Wed, 03 Oct 2007 07:23:57 GMT
Hello all,

I'm using VFS to download and upload files from/to my machines. This is 
the code I use for uploading a file.

    FileSystemOptions fsOptions = new FileSystemOptions();
        try {
            
SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsOptions, 
"no");
           
            FileSystemManager fsManager = VFS.getManager();
            String uri = "sftp://usr:pass@10.10.4.130/work/newfile";    
       
            FileObject file = fsManager.resolveFile(uri, fsOptions);    
       
            FileObject localFile = 
fsManager.resolveFile("d:\\lost.yuv.264.yuv");
            file.copyFrom(localFile, Selectors.SELECT_SELF);
    } catch (FileSystemException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

When I try to upload a file larger than 20MB, I get an out of memory 
exception:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.io.ByteArrayOutputStream.write(Unknown Source)
    at java.io.BufferedOutputStream.write(Unknown Source)
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
    at java.io.BufferedOutputStream.flush(Unknown Source)
    at java.io.FilterOutputStream.close(Unknown Source)
    at 
org.apache.commons.vfs.util.MonitorOutputStream.close(MonitorOutputStream.java:53)
    at 
org.apache.commons.vfs.provider.DefaultFileContent$FileContentOutputStream.close(DefaultFileContent.java:595)
    at org.apache.commons.vfs.FileUtil.copyContent(FileUtil.java:107)
    at 
org.apache.commons.vfs.provider.AbstractFileObject.copyFrom(AbstractFileObject.java:893)
    at SFTPTest.upload(SFTPTest.java:107)

I know I can increase the Heap size of the virtual machine, but I don't 
want to do that because my files can become extremely big (>1GB).

Is there any other way to do SFTP without the transferred data being 
stored in memory. Personally I think this is a bug, because transferring 
files through ftp should not store them in memory but flushing them now 
and then.

regards,

-- 

Nick Vercammen

Ghent University - IBCN - IBBT 
Department of Information Technology
Gaston Crommenlaan 8 bus 201
9050 Gent, Belgium
E-mail: nick.vercammen@intec.UGent.be
Tel. +32 9 33 14978
Fax +32 9 33 14899

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message