commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmar...@comcast.net
Subject Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0
Date Sat, 10 Jan 2015 01:04:56 GMT
Bernd, 

To answer your question about the HDFS version, I went back and looked at the patches I supplied
for VFS-530. In the patches that I supplied to bump the Hadoop version to 2.4.0 and 2.6.0
the only changes were in the pom. This should mean that, for the parts of the Hadoop client
objects that I am using, they are API compatible. 

For the example, the list of jars depends on the Hadoop version. I think one way to handle
that is to use the `hadoop classpath` command to create the classpath. The example would then
look like: 

REP=~/.m2/repository 
HADOOP_HOME=<PATH TO HADOOP> 
HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` 
LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar 
LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar

LIBS=$LIBS:$HADOOP_CLASSPATH 
java -cp $LIBS org.apache.commons.vfs2.example.Shell 


On my local machine, this looks like: 

08:03:17 ~/EclipseWorkspace/commons-vfs2-project$ unset REP 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_HOME 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ unset LIBS 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ REP=~/.m2/repository 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_HOME=/home/dave/Software/hadoop-2.6.0

08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop
classpath` 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar

08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar

08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:$HADOOP_CLASSPATH 
08:03:18 ~/EclipseWorkspace/commons-vfs2-project$ java -cp $LIBS org.apache.commons.vfs2.example.Shell

15/01/09 20:03:18 INFO impl.StandardFileSystemManager: Using "/tmp/vfs_cache" as temporary
files store. 
VFS Shell 2.1-SNAPSHOT 
> info 
Default manager: "org.apache.commons.vfs2.impl.StandardFileSystemManager" version 2.1-SNAPSHOT

Provider Schemes: [https, res, gz, hdfs, sftp, ftps, ram, http, file, ftp, tmp, bz2] 
Virtual Schemes: [zip, war, par, ear, jar, sar, ejb3, tar, tbz2, tgz] 


Is this sufficient? 

Dave 

----- Original Message -----

From: "Bernd Eckenfels" <ecki@zusammenkunft.net> 
To: dlmarion@comcast.net 
Cc: "Commons Developers List" <dev@commons.apache.org> 
Sent: Friday, January 9, 2015 9:05:55 AM 
Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0 

Hello Dave, 

for the download page (staged: 
http://people.apache.org/~ecki/commons-vfs/download.html) I need a list 
of libraries needed to run VFS to access hdfs. Could you maybe produce 
an example VFS Shell session similiar to 

https://wiki.apache.org/commons/VfsReleaseState 

where you list the command line needed. It would be best if this is an 
public HDFS instance (but I guess there is no such thing?) 

BTW: I had asked in the VFS-530 bug about how commonplace the different 
hdfs APIs are, and if it is really good if we bump the minimum version. 
As I understand it. Will HDFS with 1.1.2 be able to communicate with 
2.6 instances? If yes I would prefer to keep it at 1.2 in this release. 
WDYT? 

Gruss 
Bernd 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message