commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Gregory <garydgreg...@gmail.com>
Subject Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access?
Date Fri, 11 Apr 2014 19:57:30 GMT
Curious: What OS are you on?

We build trunk with Hadoop version 1.1.2, so who know what happens with a
newer version.

I would start with trying to build VFS with the version of Hadoop you want
to use to see what happens...

Gary


On Fri, Apr 11, 2014 at 3:51 PM, Roger Whitcomb
<Roger.Whitcomb@actian.com>wrote:

> Hi Martin,
> Well, I downloaded Hadoop 2.3.0 from one of the mirrors, and cannot find a
> "hadoop-core....jar" file anywhere in that distribution.  But I was able to
> track down sort of the minimum set of .jars necessary to at least (try to)
> connect using Commons VFS 2.1:
> commons-collections-3.2.1.jar
> commons-configuration-1.6.jar
> commons-lang-2.6.jar
> commons-vfs2-2.1-SNAPSHOT.jar
> guava-11.0.2.jar
> hadoop-auth-2.3.0.jar
> hadoop-common-2.3.0.jar
> log4j-1.2.17.jar
> slf4j-api-1.7.5.jar
> slf4j-log4j12-1.7.5.jar
>
> What's happening now is that I instantiated the HdfsProvider this way:
>         private static DefaultFileSystemManager manager = null;
>
>         static
>         {
>             manager = new DefaultFileSystemManager();
>             try {
>                 manager.setFilesCache(new DefaultFilesCache());
>                 manager.addProvider("hdfs", new HdfsFileProvider());
>                 manager.setFileContentInfoFactory(new
> FileContentInfoFilenameFactory());
>                 manager.setFilesCache(new SoftRefFilesCache());
>                 manager.setReplicator(new DefaultFileReplicator());
>                 manager.setCacheStrategy(CacheStrategy.ON_RESOLVE);
>                 manager.init();
>             }
>             catch (final FileSystemException e) {
>                 throw new
> RuntimeException(Intl.getString("object#manager.setupError"), e);
>             }
>         }
>
> Then, I try to browse into an HDFS system this way:
>             String url = String.format("hdfs://%1$s:%2$d/%3$s",
> "hadoop-master ", 50070, hdfsPath);
>             return manager.resolveFile(url);
>
> Which results in a bunch of error messages in the log file, which looks
> like it is trying to do user validation on the local machine instead of
> against the Hadoop (remote) cluster.
> Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG
> FileObjectManager: Trying to resolve file reference
> 'hdfs://hadoop-master:50070/'
> Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26)  INFO
> org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Apr 11,2014 18:27:39.078 GMT T[AWT-EventQueue-0](26) DEBUG
> MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate
> of successful kerberos logins and latency (milliseconds)], about=,
> type=DEFAULT, always=false, sampleName=Ops)
> Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate
> of failed kerberos logins and latency (milliseconds)], about=,
> type=DEFAULT, always=false, sampleName=Ops)
> Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with
> annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,
> value=[GetGroups], about=, type=DEFAULT, always=false, sampleName=Ops)
> Apr 11,2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> MetricsSystemImpl: UgiMetrics, User and group related metrics
> Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups:
>  Creating new Groups object
> Apr 11,2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG
> NativeCodeLoader: Trying to load the custom-built native-hadoop library...
> Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG
> NativeCodeLoader: Failed to load native-hadoop with error:
> java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
> Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG
> NativeCodeLoader: java.library.path=.... <bunch of stuff>
> Apr 11,2014 18:27:39.360 GMT T[AWT-EventQueue-0](26)  WARN
> NativeCodeLoader: Unable to load native-hadoop library for your platform...
> using builtin-java classes where applicable
> Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG
> JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
> Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG
> JniBasedUnixGroupsMappingWithFallback: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
> Apr 11,2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed
> to detect a valid hadoop home directory: HADOOP_HOME or hadoop.home.dir are
> not set.
> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
>         at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
>         at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290)
>         at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
>         at
> org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92)
>         at org.apache.hadoop.security.Groups.<init>(Groups.java:76)
>         at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:239)
>         at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
>         at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
>         at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
>         at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
>         at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
>         at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473)
>         at
> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
>         at
> org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115)
>         at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84)
>         at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:609)
>
> Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell: Failed
> to locate the winutils binary in the hadoop binary path: Could not locate
> executable null\bin\winutils.exe in the Hadoop binaries.
> java.io.IOException: Could not locate executable null\bin\winutils.exe in
> the Hadoop binaries.
>
> Apr 11,2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups: Group
> mapping
> impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback;
> cacheTimeout=300000; warningDeltaMs=5000
> Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG
> UserGroupInformation: hadoop login
> Apr 11,2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG
> UserGroupInformation: hadoop login commit
> Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG
> UserGroupInformation: using local user:NTUserPrincipal: <user_name>
> Apr 11,2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG
> UserGroupInformation: UGI loginUser:whiro01 (auth:SIMPLE)
> Apr 11,2014 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem:
> Error connecting to filesystem hdfs://hadoop-master:50070/: No FileSystem
> for scheme: hdfs
> java.io.IOException: No FileSystem for scheme: hdfs
>         at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
>         at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
>         at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
>         at
> org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFileSystem.java:115)
>         at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:84)
>         at
> org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:64)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:700)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:656)
>         at
> org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:609)
>
> So, my guess is that I don't have enough configuration setup on my client
> machine to tell Hadoop that the authentication is to be done at the remote
> end ....??  So, I'm trying to track down what the configuration info might
> be.
>
> Just hoping there are some Hadoop users/experts here who might be able to
> help.  Or maybe I need to talk to Hadoop Users list.  Just thought since I
> was using Commons VFS that someone here could help (as in the author of the
> HdfsProvider ....)
>
> Note: I want to build a GUI component that can browse to arbitrary HDFS
> installations, so I can't really be setting up a hard-coded XML file for
> each potential Hadoop cluster I might connect to ....
>
> Thanks,
> ~Roger
>
> -----Original Message-----
> From: Martin Gainty [mailto:mgainty@hotmail.com]
> Sent: Tuesday, April 08, 2014 5:56 PM
> To: Commons Users List
> Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS
> access?
>
> Roger-
>
>
>
> look for <artifactId>hadoop-core</artifactId>
> in the <dependencies><dependency> section in your pom.xml
>
> if  either
>
> >mvn compile
>
> >mvn package
> cannot locate hadoop-core
> you'll need to sync your <distributionManagement> section in pom.xml for
>
> hadoop-core snapshots
> hadoop-core releases
>
>
> Ping back if you're calling no-joy
>
> M-
>
>
>
>
> > From: Roger.Whitcomb@actian.com
> > To: user@commons.apache.org
> > Subject: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access?
> > Date: Tue, 8 Apr 2014 21:46:09 +0000
> >
> > Hi all,
> > I've written a Commons VFS file browser using Apache Pivot (it's part of
> the 2.1 upcoming release). Now I'm trying to use it to browse into HDFS
> files. I got it to work with HDFS 1.x by simple including the hadoop-core
> .jar and a couple of other commons .jars that are also needed. But, that
> approach doesn't work for Hadoop 2.x. Specifically the
> "hadoop-core-xxx.jar" doesn't exist anymore, and substituting the
> "hadoop-hdfs-xxxx.jar" doesn't work at all. So, before I bang my head
> against the wall, does anyone here know what .jars I would need for Hadoop
> 2.x to interface with the VFS HDFS provider?
> >
> > Thanks,
> > ~Roger Whitcomb
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>


-- 
E-Mail: garydgregory@gmail.com | ggregory@apache.org
Java Persistence with Hibernate, Second Edition<http://www.manning.com/bauer3/>
JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
Spring Batch in Action <http://www.manning.com/templier/>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message