Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of
 roger.whitcomb@actian.com)
From: Roger Whitcomb <Roger.Whitcomb@actian.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Which Hadoop 2.x .jars are necessary for Apache Commons VFS HDFS
 access?
Thread-Topic: Which Hadoop 2.x .jars are necessary for Apache Commons VFS
 HDFS access?
Thread-Index: Ac9VwunSVzyIcSZzSDGk1lFL4fOhPg==
Date: Fri, 11 Apr 2014 20:20:06 +0000
Message-ID: 
 <31b89ec8be3242e78125820227806b42@BN1PR06MB405.namprd06.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
MIME-Version: 1.0
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable

Hi,
I'm fairly new to Hadoop, but not to Apache, and I'm having a newbie kind o=
f issue browsing HDFS files.  I have written an Apache Commons VFS (Virtual=
 File System) browser for the Apache Pivot GUI framework (I'm the PMC Chair=
 for Pivot: full disclosure).  And now I'm trying to get this browser to wo=
rk with HDFS to do HDFS browsing from our application.  I'm running into a =
problem, which seems sort of basic, so I thought I'd ask here...

So, I downloaded Hadoop 2.3.0 from one of the mirrors, and was able to trac=
k down sort of the minimum set of .jars necessary to at least (try to) conn=
ect using Commons VFS 2.1:
commons-collections-3.2.1.jar
commons-configuration-1.6.jar
commons-lang-2.6.jar
commons-vfs2-2.1-SNAPSHOT.jar
guava-11.0.2.jar
hadoop-auth-2.3.0.jar
hadoop-common-2.3.0.jar
log4j-1.2.17.jar
slf4j-api-1.7.5.jar
slf4j-log4j12-1.7.5.jar

What's happening now is that I instantiated the HdfsProvider this way:
=09private static DefaultFileSystemManager manager =3D null;

=09static
=09{
=09    manager =3D new DefaultFileSystemManager();
=09    try {
=09=09manager.setFilesCache(new DefaultFilesCache());
=09=09manager.addProvider("hdfs", new HdfsFileProvider());
=09=09manager.setFileContentInfoFactory(new FileContentInfoFilenameFactory(=
));
=09=09manager.setFilesCache(new SoftRefFilesCache());
=09=09manager.setReplicator(new DefaultFileReplicator());
=09=09manager.setCacheStrategy(CacheStrategy.ON_RESOLVE);
=09=09manager.init();
=09    }
=09    catch (final FileSystemException e) {
=09=09throw new RuntimeException(Intl.getString("object#manager.setupError"=
), e);
=09    }
=09}

Then, I try to browse into an HDFS system this way:
=09    String url =3D String.format("hdfs://%1$s:%2$d/%3$s", "hadoop-master=
 ", 50070, hdfsPath);
=09    return manager.resolveFile(url);

Note: the client is running on Windows 7 (but could be any system that runs=
 Java), and the target has been one of several Hadoop clusters on Ubuntu VM=
s (basically the same thing happens no matter which Hadoop installation I t=
ry to hit).  So I'm guessing the problem is in my client configuration.

This attempt to basically just connect to HDFS results in a bunch of error =
messages in the log file, which looks like it is trying to do user validati=
on on the local machine instead of against the Hadoop (remote) cluster.
Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG FileObjectManage=
r: Trying to resolve file reference 'hdfs://hadoop-master:50070/'
Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26)  INFO org.apache.hadoo=
p.conf.Configuration.deprecation: fs.default.name is deprecated. Instead, u=
se fs.defaultFS
Apr 11,2014 18:27:39.078 GMT T[AWT-EventQueue-0](26) DEBUG MutableMetricsFa=
ctory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.s=
ecurity.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.a=
pache.hadoop.metrics2.annotation.Metric(valueName=3DTime, value=3D[Rate of =
successful kerberos logins and latency (milliseconds)], about=3D, type=3DDE=
FAULT, always=3Dfalse, sampleName=3DOps)
Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG MutableMetricsFa=
ctory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.s=
ecurity.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.a=
pache.hadoop.metrics2.annotation.Metric(valueName=3DTime, value=3D[Rate of =
failed kerberos logins and latency (milliseconds)], about=3D, type=3DDEFAUL=
T, always=3Dfalse, sampleName=3DOps)
Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG MutableMetricsFa=
ctory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.s=
ecurity.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apac=
he.hadoop.metrics2.annotation.Metric(valueName=3DTime, value=3D[GetGroups],=
 about=3D, type=3DDEFAULT, always=3Dfalse, sampleName=3DOps)
Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG MetricsSystemImp=
l: UgiMetrics, User and group related metrics
Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups:  Creatin=
g new Groups object
Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG NativeCodeLoader=
: Trying to load the custom-built native-hadoop library...
Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG NativeCodeLoader=
: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: =
no hadoop in java.library.path
Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG NativeCodeLoader=
: java.library.path=3D.... <bunch of stuff>
Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26)  WARN NativeCodeLoader=
: Unable to load native-hadoop library for your platform... using builtin-j=
ava classes where applicable
Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG JniBasedUnixGrou=
psMappingWithFallback: Falling back to shell based
Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG JniBasedUnixGrou=
psMappingWithFallback: Group mapping impl=3Dorg.apache.hadoop.security.Shel=
lBasedUnixGroupsMapping
Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed to=
 detect a valid hadoop home directory: HADOOP_HOME or hadoop.home.dir are n=
ot set.
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
=09at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
=09at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290)
=09at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
=09at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92)
=09at org.apache.hadoop.security.Groups.<init>(Groups.java:76)
=09at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Group=
s.java:239)
=09at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupI=
nformation.java:255)
=09at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(Use=
rGroupInformation.java:232)
=09at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(=
UserGroupInformation.java:718)
=09at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGrou=
pInformation.java:703)
=09at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGr=
oupInformation.java:605)
=09at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473=
)
=09at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465=
)
=09at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331)
=09at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
=09at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
=09at org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(Hdfs=
FileSystem.java:115)
=09at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.find=
File(AbstractOriginatingFileProvider.java:84)
=09at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.find=
File(AbstractOriginatingFileProvider.java:64)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:700)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:656)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:609)

Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed to=
 locate the winutils binary in the hadoop binary path: Could not locate exe=
cutable null\bin\winutils.exe in the Hadoop binaries.
java.io.IOException: Could not locate executable null\bin\winutils.exe in t=
he Hadoop binaries.

Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups: Group ma=
pping impl=3Dorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallba=
ck; cacheTimeout=3D300000; warningDeltaMs=3D5000
Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG UserGroupInforma=
tion: hadoop login
Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG UserGroupInforma=
tion: hadoop login commit
Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG UserGroupInforma=
tion: using local user:NTUserPrincipal: <user_name>
Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG UserGroupInforma=
tion: UGI loginUser:whiro01 (auth:SIMPLE)
Apr 11,2014 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem: =
Error connecting to filesystem hdfs://hadoop-master:50070/: No FileSystem f=
or scheme: hdfs
java.io.IOException: No FileSystem for scheme: hdfs
=09at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:23=
04)
=09at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311=
)
=09at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
=09at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:235=
0)
=09at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
=09at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
=09at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
=09at org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(Hdfs=
FileSystem.java:115)
=09at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.find=
File(AbstractOriginatingFileProvider.java:84)
=09at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.find=
File(AbstractOriginatingFileProvider.java:64)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:700)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:656)
=09at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
aultFileSystemManager.java:609)

So, my guess is that I don't have enough configuration setup on my client m=
achine to tell Hadoop that the authentication is to be done at the remote e=
nd ....??  So, I'm trying to track down what the configuration info might b=
e.

Hoping to see if anyone here can see past the Commons VFS stuff that you pr=
obably don't understand to be able to tell me what other Hadoop/HDFS files =
/ configuration I need to get this working.

Note: I want to build a GUI component that can browse to arbitrary HDFS ins=
tallations, so I can't really be setting up a hard-coded XML file for each =
potential Hadoop cluster I might connect to ....

Thanks,
~Roger Whitcomb