commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Eckenfels <e...@zusammenkunft.net>
Subject Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0
Date Sat, 10 Jan 2015 01:13:27 GMT
Am Sat, 10 Jan 2015 01:04:56 +0000 (UTC)
schrieb dlmarion@comcast.net:

> To answer your question about the HDFS version, I went back and
> looked at the patches I supplied for VFS-530. In the patches that I
> supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only
> changes were in the pom. This should mean that, for the parts of the
> Hadoop client objects that I am using, they are API compatible. 

Well, you never know. But if thats the case there is also no problem if
we stay at the older version for now.

> For the example, the list of jars depends on the Hadoop version. I
> think one way to handle that is to use the `hadoop classpath` command
> to create the classpath. The example would then look like: 

Looks great, thanks.

> VFS Shell 2.1-SNAPSHOT 
> > info 
> Default manager:
> "org.apache.commons.vfs2.impl.StandardFileSystemManager" version
> 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps,
> ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear,
> jar, sar, ejb3, tar, tbz2, tgz] 
> 
> 
> Is this sufficient? 

Did you try to use ls or another command to actually connect to a
hdfs name node? I think you should be able to provide
"user:password@host" with the Shell.

I have currently the problem, that the hdfs tests fail (under Linux)
with some form of directory lock problem. I am not sure if this worked
before (as I havent tested on my windows). I will see if the version
upgrade helps, if yes I will for sure include it in 2.1

Greetings
Bernd

> 
> Dave 
> 
> ----- Original Message -----
> 
> From: "Bernd Eckenfels" <ecki@zusammenkunft.net> 
> To: dlmarion@comcast.net 
> Cc: "Commons Developers List" <dev@commons.apache.org> 
> Sent: Friday, January 9, 2015 9:05:55 AM 
> Subject: Re: [VFS] Implementing custom hdfs file system using
> commons-vfs 2.0 
> 
> Hello Dave, 
> 
> for the download page (staged: 
> http://people.apache.org/~ecki/commons-vfs/download.html) I need a
> list of libraries needed to run VFS to access hdfs. Could you maybe
> produce an example VFS Shell session similiar to 
> 
> https://wiki.apache.org/commons/VfsReleaseState 
> 
> where you list the command line needed. It would be best if this is
> an public HDFS instance (but I guess there is no such thing?) 
> 
> BTW: I had asked in the VFS-530 bug about how commonplace the
> different hdfs APIs are, and if it is really good if we bump the
> minimum version. As I understand it. Will HDFS with 1.1.2 be able to
> communicate with 2.6 instances? If yes I would prefer to keep it at
> 1.2 in this release. WDYT? 
> 
> Gruss 
> Bernd 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message