commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Whitcomb <Roger.Whitc...@actian.com>
Subject RE: [VFS] Could the HDFS connector work with MapR systems?
Date Wed, 10 Jun 2015 20:22:19 GMT
Hi Ted,
	At this point I gave up and implemented the workaround to browse the NFS mount area instead.
 The test case was kind of complicated anyway, since it involves the Pivot VFS File Browser
as we as the Commons VFS code.

Thanks,
~Roger

-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com] 
Sent: Tuesday, May 26, 2015 8:22 PM
To: Commons Users List
Subject: Re: [VFS] Could the HDFS connector work with MapR systems?

The NFS mount trick allows all of the files to appear via the local://* file system.

Otherwise, it is true that the wire protocol for MapR FS is different from HDFS.  In fact,
the wire protocol for different versions of HDFS are different as well.

But what is standard is how different implementations of a Hadoop compatible file system are
plugged into the file space.  That should work for any reasonable application if you have
the right jars in your search path.

One possibility where this would break would be if the client code assumes, for instance,
that all file paths must have a server address.  Otherwise, well-behaved HDFS client applications
should work.  The null host name exception sounds like there is a format exception going on.

Can I see the some test code and stack traces somewhere?






On Tue, May 26, 2015 at 4:32 PM, Roger Whitcomb <Roger.Whitcomb@actian.com>
wrote:

> So, I'm pretty sure (now) that it DOESN'T work.  There are comments here:
> http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting
> -to-namenode.html
> and here:
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Application
> s plus the connector here:  
> https://github.com/pentaho/pentaho-hdfs-vfs
> that all point to special MapR code being needed, and the standard 
> HDFS client protocols not working.  Plus all my testing seemed to confirm this.
>
> We have figured out a workaround for our configuration, though (we're 
> basically requiring an NFS mount of the MapR file system, so we can 
> just browse the MapR files as native).
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Gary Gregory [mailto:garydgregory@gmail.com]
> Sent: Tuesday, May 26, 2015 3:48 PM
> To: Commons Users List
> Subject: RE: [VFS] Could the HDFS connector work with MapR systems?
>
> I've not seen VFS used in this configuration before. You could try 
> updating to the current HDFS jar files.
> Gary
>
> -------- Original message --------
> From: Roger Whitcomb <Roger.Whitcomb@actian.com>
> Date: 05/26/2015  10:39  (GMT-08:00)
> To: Commons Users List <user@commons.apache.org>
> Subject: [VFS] Could the HDFS connector work with MapR systems?
>
> Hi all,
>                 I'm using Commons VFS 2.1 along with the HDFS 
> connector and everything is working brilliantly ... until, that is, 
> trying to access HDFS on a MapR system.  Then I get things like null 
> host name exceptions and the like.  So, anyone have any idea whether 
> the HDFS connector even COULD work against MapR?  And if not, anyone 
> tried to build a MapR connector to Commons VFS 2.1?
>
> Thanks in advance,
> ~Roger Whitcomb
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org

Mime
View raw message