Mailing-List: contact user-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Commons Users List" <user@commons.apache.org>
MIME-Version: 1.0
In-Reply-To: 
 <SN1PR0601MB156535BBD211AE6E65B3F736E7CC0@SN1PR0601MB1565.namprd06.prod.outlook.com>
References: <5a06gn4fny2ptrt8vet3fa0q.1432680480374@email.android.com>
 <SN1PR0601MB156535BBD211AE6E65B3F736E7CC0@SN1PR0601MB1565.namprd06.prod.outlook.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Tue, 26 May 2015 20:22:11 -0700
Message-ID: 
 <CAJwFCa0kGTUai-dwhpXm1TnCTZkVuLb-3hRRnBsntLYc18f9XA@mail.gmail.com>
Subject: Re: [VFS] Could the HDFS connector work with MapR systems?
To: Commons Users List <user@commons.apache.org>
Content-Type: multipart/alternative; boundary=001a113f280a74366d051707c1c2

--001a113f280a74366d051707c1c2
Content-Type: text/plain; charset=UTF-8

The NFS mount trick allows all of the files to appear via the local://*
file system.

Otherwise, it is true that the wire protocol for MapR FS is different from
HDFS.  In fact, the wire protocol for different versions of HDFS are
different as well.

But what is standard is how different implementations of a Hadoop
compatible file system are plugged into the file space.  That should work
for any reasonable application if you have the right jars in your search
path.

One possibility where this would break would be if the client code assumes,
for instance, that all file paths must have a server address.  Otherwise,
well-behaved HDFS client applications should work.  The null host name
exception sounds like there is a format exception going on.

Can I see the some test code and stack traces somewhere?


On Tue, May 26, 2015 at 4:32 PM, Roger Whitcomb <Roger.Whitcomb@actian.com>
wrote:

> So, I'm pretty sure (now) that it DOESN'T work.  There are comments here:
> http://answers.mapr.com/questions/9224/apache-hadoop-client-connecting-to-namenode.html
> and here:
> http://doc.mapr.com/display/MapR/Accessing+MapR-FS+in+Java+Applications
> plus the connector here:  https://github.com/pentaho/pentaho-hdfs-vfs
> that all point to special MapR code being needed, and the standard HDFS
> client protocols not working.  Plus all my testing seemed to confirm this.
>
> We have figured out a workaround for our configuration, though (we're
> basically requiring an NFS mount of the MapR file system, so we can just
> browse the MapR files as native).
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Gary Gregory [mailto:garydgregory@gmail.com]
> Sent: Tuesday, May 26, 2015 3:48 PM
> To: Commons Users List
> Subject: RE: [VFS] Could the HDFS connector work with MapR systems?
>
> I've not seen VFS used in this configuration before. You could try
> updating to the current HDFS jar files.
> Gary
>
> -------- Original message --------
> From: Roger Whitcomb <Roger.Whitcomb@actian.com>
> Date: 05/26/2015  10:39  (GMT-08:00)
> To: Commons Users List <user@commons.apache.org>
> Subject: [VFS] Could the HDFS connector work with MapR systems?
>
> Hi all,
>                 I'm using Commons VFS 2.1 along with the HDFS connector
> and everything is working brilliantly ... until, that is, trying to access
> HDFS on a MapR system.  Then I get things like null host name exceptions
> and the like.  So, anyone have any idea whether the HDFS connector even
> COULD work against MapR?  And if not, anyone tried to build a MapR
> connector to Commons VFS 2.1?
>
> Thanks in advance,
> ~Roger Whitcomb
>

--001a113f280a74366d051707c1c2--