commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Commons Wiki] Update of "SimpleSftpFileDownload" by KenTanaka
Date Tue, 25 Mar 2008 20:11:31 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Commons Wiki" for change notification.

The following page has been changed by KenTanaka:
http://wiki.apache.org/jakarta-commons/SimpleSftpFileDownload

The comment on the change is:
Added code and output

------------------------------------------------------------------------------
  = Key Concepts =
  Access to a remote system using SFTP uses the SSH secure shell protocols. Although the behavior
is similar to FTP, it is ''not'' FTP run over a secure connection. So there are some differences
between FTP and SFTP that should be noted. One of these it the lack of the FTP binary/ASCII
transfer mode, in SFTP all transfers are binary as if they were executed with an "scp" (secure
copy) command.
  
- This example code uses a regular expression to match files on the remote system, so that
not all of the files in the source directory are transferred.
+ This example code uses a regular expression to match files on the remote system, so that
not all of the files in the source directory are transferred. The {{{filePatternString}}}
is set to ".*/smoke\\d{8}_wkt\\.txt". This has the regular expression components of:
+  * ".*/" matches any path that precedes the filename. The "." is wildcard character, "*"
specifies that 0 or more of these may be present. The "/" matches the directory separator.
+  * "\\d{8}" matches 8 digits (4 for the year, 2 for the month number, and 2 for the day
of the month). The doubled backslash is to escape a single backslash for Java string literals.
"\d" indicates any digit, and "{8}" specifies exactly 8 of the preceding character or pattern.
+  * "\\." is an escaped "\." which means a literal period instead of a wild card interpretation
of "."
+ 
+ It is important to call the [http://commons.apache.org/vfs/apidocs/org/apache/commons/vfs/impl/DefaultFileSystemManager.html#close()
close()] method of the !DefaultFileSystemManager to clean up any temporary files and close
all providers. Otherwise the program will appear to hang after downloading files.
  
  = Source Code =
+ The code provided below is for a Maven 2 project.
+ 
  == pom.xml Project File ==
- This example uses Maven2. There is a '''{{{pom.xml}}}''' to define the project
+ The '''{{{pom.xml}}}''' file defines how the project is built for maven 2:
  
  {{{
+ <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+     <modelVersion>4.0.0</modelVersion>
+     <groupId>gov.noaa.eds.byExample</groupId>
+     <artifactId>trySimpleVfsSftp</artifactId>
+     <packaging>jar</packaging>
+     <version>1.0-SNAPSHOT</version>
+     <name>trySimpleVfsSftp</name>
+     <url>http://maven.apache.org</url>
+     <build>
+         <extensions>
+             <extension>
+                 <groupId>org.apache.maven.wagon</groupId>
+                 <artifactId>wagon-ssh-external</artifactId>
+                 <version>1.0-beta-2</version>
+             </extension>
+         </extensions>
+         <plugins>
+             <plugin>
+                 <artifactId>maven-compiler-plugin</artifactId>
+                 <configuration>
+                     <source>1.5</source>
+                     <target>1.5</target>
+                 </configuration>
+             </plugin>
+             <plugin>
+                 <artifactId>maven-assembly-plugin</artifactId>
+                 <configuration>
+                     <descriptorRefs>
+                         <descriptorRef>jar-with-dependencies</descriptorRef>
+                     </descriptorRefs>
+                     <archive>
+                         <manifest>
+                             <mainClass>gov.noaa.eds.byExample.trySimpleVfsSftp.App</mainClass>
+                         </manifest>
+                     </archive>
+                 </configuration>
+             </plugin>
+             <plugin>
+                 <artifactId>maven-antrun-plugin</artifactId>
+                 <configuration>
+                     <tasks>
+                         <java classname="gov.noaa.eds.byExample.trySimpleVfsSftp.App"
classpathref="maven.runtime.classpath">
+                         </java>
+                     </tasks>
+                 </configuration>
+             </plugin>
+         </plugins>
+     </build>
+     <dependencies>
+         <!-- Supports VFS SFTP -->
+         <dependency>
+             <groupId>com.jcraft</groupId>
+             <artifactId>jsch</artifactId>
+             <version>0.1.23</version>
+             <optional>true</optional>
+         </dependency>
+         <dependency>
+             <groupId>commons-vfs</groupId>
+             <artifactId>commons-vfs</artifactId>
+             <version>1.0</version>
+         </dependency>
+         <dependency>
+             <groupId>junit</groupId>
+             <artifactId>junit</artifactId>
+             <version>3.8.1</version>
+             <scope>test</scope>
+         </dependency>
+     </dependencies>
+ </project>
  }}}
  
  
@@ -32, +109 @@

  /*
   * App.java
   */
+ package gov.noaa.eds.byExample.trySimpleVfsSftp;
+ 
+ import java.io.File;
+ import java.util.regex.Pattern;
+ import org.apache.commons.vfs.AllFileSelector;
+ import org.apache.commons.vfs.FileObject;
+ import org.apache.commons.vfs.FileSystemException;
+ import org.apache.commons.vfs.FileSystemManager;
+ import org.apache.commons.vfs.FileSystemOptions;
+ import org.apache.commons.vfs.FileType;
+ import org.apache.commons.vfs.UserAuthenticator;
+ import org.apache.commons.vfs.VFS;
+ import org.apache.commons.vfs.auth.StaticUserAuthenticator;
+ import org.apache.commons.vfs.impl.DefaultFileSystemConfigBuilder;
+ import org.apache.commons.vfs.impl.DefaultFileSystemManager;
+ import org.apache.commons.vfs.impl.StandardFileSystemManager;
+ import org.apache.commons.vfs.provider.local.LocalFile;
+ 
+ 
+ /**
+  * Example use of VFS sftp
+  *
+  */
+ public class App {
+ 
+     // Set these variables for your testing environment:
+     private String host = "sftpremote.example.com";  // Remote SFTP hostname
+     private String user = "smokey";      // Remote system login name
+     private String password = "bear";    // Remote system password
+     private String remoteDir = "/data/source/fires/smoke";
+     // Look for a file path like "smoke20070128_wkt.txt"
+     private String filePatternString = ".*/smoke\\d{8}_wkt\\.txt";
+     // Local directory to receive file
+     private String localDir = "/extra/data/fires/smoke";
+     
+     
+     private File localDirFile;
+     private Pattern filePattern;
+     private FileSystemManager fsManager = null;
+     private FileSystemOptions opts = null;
+     private FileObject sftpFile;
+ 
+ 
+     public static void main(String[] args) {
+         System.out.println("SFTP download");
+         App app = new App();
+ 
+         app.initialize();
+ 
+         app.process();
+ 
+         app.release();
+ 
+     } // main( String[] args )
+ 
+ 
+     /**
+      * Creates the download directory localDir if it
+      * does not exist and makes a connection to the remote SFTP server.
+      * 
+      */
+     public void initialize() {
+         if (localDirFile == null) {
+             localDirFile = new File(localDir);
+         }
+         if (!this.localDirFile.exists()) {
+             localDirFile.mkdirs();
+         }
+ 
+         try {
+             this.fsManager = VFS.getManager();
+         } catch (FileSystemException ex) {
+             throw new RuntimeException("failed to get fsManager from VFS", ex);
+         }
+ 
+         UserAuthenticator auth = new StaticUserAuthenticator(null, this.user,
+                 this.password);
+         this.opts = new FileSystemOptions();
+         try {
+             DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(opts,
+                     auth);
+         } catch (FileSystemException ex) {
+             throw new RuntimeException("setUserAuthenticator failed", ex);
+         }
+ 
+         this.filePattern = Pattern.compile(filePatternString);
+     } // initialize()
+ 
+ 
+     /**
+      * Retrieves files that match the specified FileSpec from the SFTP server
+      * and stores them in the local directory.
+      */
+     public void process() {
+ 
+         String startPath = "sftp://" + this.host + this.remoteDir;
+         FileObject[] children;
+ 
+         // Set starting path on remote SFTP server.
+         try {
+             this.sftpFile = this.fsManager.resolveFile(startPath, opts);
+ 
+             System.out.println("SFTP connection successfully established to " +
+                     startPath);
+         } catch (FileSystemException ex) {
+             throw new RuntimeException("SFTP error parsing path " +
+                     this.remoteDir,
+                     ex);
+         }
+ 
+ 
+         // Get a directory listing
+         try {
+             children = this.sftpFile.getChildren();
+         } catch (FileSystemException ex) {
+             throw new RuntimeException("Error collecting directory listing of " +
+                     startPath, ex);
+         }
+ 
+         search:
+         for (FileObject f : children) {
+             try {
+                 String relativePath =
+                         File.separatorChar + f.getName().getBaseName();
+ 
+                 if (f.getType() == FileType.FILE) {
+                     System.out.println("Examining remote file " + f.getName());
+ 
+                     if (!this.filePattern.matcher(f.getName().getPath()).matches()) {
+                         System.out.println("  Filename does not match, skipping file ."
+
+                                 relativePath);
+                         continue search;
+                     }
+ 
+                     String localUrl = "file://" + this.localDir + relativePath;
+                     String standardPath = this.localDir + relativePath;
+                     System.out.println("  Standard local path is " + standardPath);
+                     LocalFile localFile =
+                             (LocalFile) this.fsManager.resolveFile(localUrl);
+                     System.out.println("    Resolved local file name: " +
+                             localFile.getName());
+ 
+                     if (!localFile.getParent().exists()) {
+                         localFile.getParent().createFolder();
+                     }
+ 
+                     System.out.println("  ### Retrieving file ###");
+                     localFile.copyFrom(f,
+                             new AllFileSelector());
+                 } else {
+                     System.out.println("Ignoring non-file " + f.getName());
+                 }
+             } catch (FileSystemException ex) {
+                 throw new RuntimeException("Error getting file type for " +
+                         f.getName(), ex);
+             }
+         } // for (FileObject f : children)
+     } // process(Object obj)
+ 
+ 
+     public void release() {
+         ((DefaultFileSystemManager) this.fsManager).close();
+     } // release()
+ } // class App
  }}}
  
  = Compiling =
@@ -48, +289 @@

  
  == Sample Output ==
  {{{
- 
+ SFTP download
+ Mar 25, 2008 1:00:44 PM org.apache.commons.vfs.VfsLog info
+ INFO: Using "/tmp/vfs_cache" as temporary files store.
+ SFTP connection successfully established to sftp://sftpremote.example.com/data/source/fires/smoke
+ Examining remote file sftp://sftpremote.example.com/data/source/fires/smoke/README.txt
+   Filename does not match, skipping file ./README.txt
+ Examining remote file sftp://sftpremote.example.com/data/source/fires/smoke/smoke20070328_wkt.txt
+   Standard local path is /extra/data/fires/smoke/smoke20070328_wkt.txt
+     Resolved local file name: file:///extra/data/fires/smoke/smoke20070328_wkt.txt
+   ### Retrieving file ###
+ Examining remote file sftp://sftpremote.example.com/data/source/fires/smoke/smoke20070329.txt
+   Filename does not match, skipping file ./smoke20070329.txt
  }}}
  There should now be files matching the {{{filePatternString}}} in the local machine directory
"/extra/data/fires/smoke".
  

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message