Mailing-List: contact user-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Commons Users List" <user@commons.apache.org>
Received-SPF: pass (athena.apache.org: domain of mgainty@hotmail.com
 designates 65.55.116.99 as permitted sender)
Message-ID: <BLU172-W50760388571FF03346E414AE570@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_15e7f6c3-a799-41be-a354-cf66a520f219_"
From: Martin Gainty <mgainty@hotmail.com>
To: "user@commons.apache.org" <user@commons.apache.org>
Subject: RE: VFS HDFS access
Date: Sat, 12 Apr 2014 09:41:02 -0400
Importance: Normal
In-Reply-To: <0scL9NtDE000233a2@blu0-omc3-s15.blu0.hotmail.com>
References: <0scL9NtDE000233a2@blu0-omc3-s15.blu0.hotmail.com>
MIME-Version: 1.0

--_15e7f6c3-a799-41be-a354-cf66a520f219_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

--Forwarded Message Attachment--
From: mgainty@hotmail.com
To: user@commons.apache.org
Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS acces=
s?
Date: Sat=2C 12 Apr 2014 08:09:25 -0400


Morning Roger and Gary
=20
Brandon Li figured this out last year by installing NFS on Wondows and then=
 mounting a HDFS mountpoint

http://lucene.472066.n3.nabble.com/Mounting-HDFS-on-client-td4107671.html
=20
Sorry for the long delay in response..Ive been knee deep in multiple house =
repair items..this has been a long costly winter

HTH
Martin Gainty=20
______________________________________________=20
Verzicht und Vertraulichkeitanmerkung/Note de d=E9ni et de confidentialit=
=E9

Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaeng=
er sein=2C so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiter=
leitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient l=
ediglich dem Austausch von Informationen und entfaltet keine rechtliche Bin=
dungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen w=
ir keine Haftung fuer den Inhalt uebernehmen.

Ce message est confidentiel et peut =EAtre privil=E9gi=E9. Si vous n'=EAtes=
 pas le destinataire pr=E9vu=2C nous te demandons avec bont=E9 que pour sat=
isfaire informez l'exp=E9diteur. N'importe quelle diffusion non autoris=E9e=
 ou la copie de ceci est interdite. Ce message sert =E0 l'information seule=
ment et n'aura pas n'importe quel effet l=E9galement obligatoire. =C9tant d=
onn=E9 que les email peuvent facilement =EAtre sujets =E0 la manipulation=
=2C nous ne pouvons accepter aucune responsabilit=E9 pour le contenu fourni=
.

 =20

> Date: Fri=2C 11 Apr 2014 17:49:10 -0400
> Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS acc=
ess?
> From: garydgregory@gmail.com
> To: user@commons.apache.org
>=20
> FWIW=2C I've not had luck getting VFS with Hadoop 1.1.2 to work on Window=
s. I
> run the unit tests on a Linux VM. YMMV.
>=20
> Gary
>=20
>=20
> On Fri=2C Apr 11=2C 2014 at 5:07 PM=2C Roger Whitcomb
> <Roger.Whitcomb@actian.com>wrote:
>=20
> > My client is on Windows 7=2C but the servers (Hadoop clusters) are most=
ly
> > Ubuntu (Cloudera VMs) for now.
> >
> > There doesn't seem to be any problem of Hadoop versions (i.e.=2C it fin=
ds
> > and loads the Hadoop classes just fine)=2C but it isn't being told what=
 the
> > right configuration is.
> >
> > I've now seem some other postings elsewhere that seem to indicate that =
the
> > FS.defaultFS setting in the Configuration object has to point to the re=
mote
> > system. So=2C I'm guessing that the VFS HdfsProvider needs to know some=
 more
> > than I am telling it. Or maybe this is a change from Hadoop 1.x to 2.x?=
!
> >
> > Thanks=2C
> > ~Roger
> >
> > -----Original Message-----
> > From: Gary Gregory [mailto:garydgregory@gmail.com]
> > Sent: Friday=2C April 11=2C 2014 12:58 PM
> > To: Commons Users List
> > Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS
> > access?
> >
> > Curious: What OS are you on?
> >
> > We build trunk with Hadoop version 1.1.2=2C so who know what happens wi=
th a
> > newer version.
> >
> > I would start with trying to build VFS with the version of Hadoop you w=
ant
> > to use to see what happens...
> >
> > Gary
> >
> >
> > On Fri=2C Apr 11=2C 2014 at 3:51 PM=2C Roger Whitcomb
> > <Roger.Whitcomb@actian.com>wrote:
> >
> > > Hi Martin=2C
> > > Well=2C I downloaded Hadoop 2.3.0 from one of the mirrors=2C and cann=
ot
> > > find a "hadoop-core....jar" file anywhere in that distribution. But I
> > > was able to track down sort of the minimum set of .jars necessary to
> > > at least (try to) connect using Commons VFS 2.1:
> > > commons-collections-3.2.1.jar
> > > commons-configuration-1.6.jar
> > > commons-lang-2.6.jar
> > > commons-vfs2-2.1-SNAPSHOT.jar
> > > guava-11.0.2.jar
> > > hadoop-auth-2.3.0.jar
> > > hadoop-common-2.3.0.jar
> > > log4j-1.2.17.jar
> > > slf4j-api-1.7.5.jar
> > > slf4j-log4j12-1.7.5.jar
> > >
> > > What's happening now is that I instantiated the HdfsProvider this way=
:
> > > private static DefaultFileSystemManager manager =3D null=3B
> > >
> > > static
> > > {
> > > manager =3D new DefaultFileSystemManager()=3B
> > > try {
> > > manager.setFilesCache(new DefaultFilesCache())=3B
> > > manager.addProvider("hdfs"=2C new HdfsFileProvider())=3B
> > > manager.setFileContentInfoFactory(new
> > > FileContentInfoFilenameFactory())=3B
> > > manager.setFilesCache(new SoftRefFilesCache())=3B
> > > manager.setReplicator(new DefaultFileReplicator())=3B
> > > manager.setCacheStrategy(CacheStrategy.ON_RESOLVE)=3B
> > > manager.init()=3B
> > > }
> > > catch (final FileSystemException e) {
> > > throw new
> > > RuntimeException(Intl.getString("object#manager.setupError")=2C e)=3B
> > > }
> > > }
> > >
> > > Then=2C I try to browse into an HDFS system this way:
> > > String url =3D String.format("hdfs://%1$s:%2$d/%3$s"=2C
> > > "hadoop-master "=2C 50070=2C hdfsPath)=3B
> > > return manager.resolveFile(url)=3B
> > >
> > > Which results in a bunch of error messages in the log file=2C which
> > > looks like it is trying to do user validation on the local machine
> > > instead of against the Hadoop (remote) cluster.
> > > Apr 11=2C2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG
> > > FileObjectManager: Trying to resolve file reference
> > > 'hdfs://hadoop-master:50070/'
> > > Apr 11=2C2014 18:27:38.953 GMT T[AWT-EventQueue-0](26) INFO
> > > org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is
> > > deprecated. Instead=2C use fs.defaultFS Apr 11=2C2014 18:27:39.078 GM=
T
> > > T[AWT-EventQueue-0](26) DEBUG
> > > MutableMetricsFactory: field
> > > org.apache.hadoop.metrics2.lib.MutableRate
> > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSucce=
s
> > > s
> > > with annotation
> > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=3DTime=2C
> > > value=3D[Rate of successful kerberos logins and latency (milliseconds=
)]=2C
> > > about=3D=2C type=3DDEFAULT=2C always=3Dfalse=2C sampleName=3DOps) Apr=
 11=2C2014
> > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> > > MutableMetricsFactory: field
> > > org.apache.hadoop.metrics2.lib.MutableRate
> > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailu=
r
> > > e
> > > with annotation
> > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=3DTime=2C
> > > value=3D[Rate of failed kerberos logins and latency (milliseconds)]=
=2C
> > > about=3D=2C type=3DDEFAULT=2C always=3Dfalse=2C sampleName=3DOps) Apr=
 11=2C2014
> > > 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> > > MutableMetricsFactory: field
> > > org.apache.hadoop.metrics2.lib.MutableRate
> > > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups
> > > with annotation
> > > @org.apache.hadoop.metrics2.annotation.Metric(valueName=3DTime=2C
> > > value=3D[GetGroups]=2C about=3D=2C type=3DDEFAULT=2C always=3Dfalse=
=2C sampleName=3DOps)
> > > Apr 11=2C2014 18:27:39.094 GMT T[AWT-EventQueue-0](26) DEBUG
> > > MetricsSystemImpl: UgiMetrics=2C User and group related metrics Apr
> > > 11=2C2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG Groups:
> > > Creating new Groups object
> > > Apr 11=2C2014 18:27:39.344 GMT T[AWT-EventQueue-0](26) DEBUG
> > > NativeCodeLoader: Trying to load the custom-built native-hadoop
> > library...
> > > Apr 11=2C2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG
> > > NativeCodeLoader: Failed to load native-hadoop with error:
> > > java.lang.UnsatisfiedLinkError: no hadoop in java.library.path Apr
> > > 11=2C2014 18:27:39.360 GMT T[AWT-EventQueue-0](26) DEBUG
> > > NativeCodeLoader: java.library.path=3D.... <bunch of stuff> Apr 11=2C=
2014
> > > 18:27:39.360 GMT T[AWT-EventQueue-0](26) WARN
> > > NativeCodeLoader: Unable to load native-hadoop library for your
> > platform...
> > > using builtin-java classes where applicable Apr 11=2C2014 18:27:39.37=
5
> > > GMT T[AWT-EventQueue-0](26) DEBUG
> > > JniBasedUnixGroupsMappingWithFallback: Falling back to shell based Ap=
r
> > > 11=2C2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) DEBUG
> > > JniBasedUnixGroupsMappingWithFallback: Group mapping
> > > impl=3Dorg.apache.hadoop.security.ShellBasedUnixGroupsMapping
> > > Apr 11=2C2014 18:27:39.375 GMT T[AWT-EventQueue-0](26) ERROR Shell:
> > > Failed to detect a valid hadoop home directory: HADOOP_HOME or
> > > hadoop.home.dir are not set.
> > > java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
> > > at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
> > > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:290)
> > > at
> > org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> > > at
> > > org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:92)
> > > at org.apache.hadoop.security.Groups.<init>(Groups.java:76)
> > > at
> > >
> > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.=
java:239)
> > > at
> > >
> > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInf=
ormation.java:255)
> > > at
> > >
> > org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserG=
roupInformation.java:232)
> > > at
> > >
> > org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(Us=
erGroupInformation.java:718)
> > > at
> > >
> > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupI=
nformation.java:703)
> > > at
> > >
> > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGrou=
pInformation.java:605)
> > > at
> > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2473=
)
> > > at
> > > org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2465=
)
> > > at
> > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2331)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
> > > at
> > >
> > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFi=
leSystem.java:115)
> > > at
> > >
> > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFi=
le(AbstractOriginatingFileProvider.java:84)
> > > at
> > >
> > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFi=
le(AbstractOriginatingFileProvider.java:64)
> > > at
> > >
> > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defau=
ltFileSystemManager.java:700)
> > > at
> > >
> > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defau=
ltFileSystemManager.java:656)
> > > at
> > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
a
> > > ultFileSystemManager.java:609)
> > >
> > > Apr 11=2C2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) ERROR Shell:
> > > Failed to locate the winutils binary in the hadoop binary path: Could
> > > not locate executable null\bin\winutils.exe in the Hadoop binaries.
> > > java.io.IOException: Could not locate executable null\bin\winutils.ex=
e
> > > in the Hadoop binaries.
> > >
> > > Apr 11=2C2014 18:27:39.391 GMT T[AWT-EventQueue-0](26) DEBUG Groups:
> > > Group mapping
> > > impl=3Dorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallba=
ck=3B
> > > cacheTimeout=3D300000=3B warningDeltaMs=3D5000 Apr 11=2C2014 18:27:39=
.469 GMT
> > > T[AWT-EventQueue-0](26) DEBUG
> > > UserGroupInformation: hadoop login
> > > Apr 11=2C2014 18:27:39.469 GMT T[AWT-EventQueue-0](26) DEBUG
> > > UserGroupInformation: hadoop login commit Apr 11=2C2014 18:27:39.751 =
GMT
> > > T[AWT-EventQueue-0](26) DEBUG
> > > UserGroupInformation: using local user:NTUserPrincipal: <user_name>
> > > Apr 11=2C2014 18:27:39.751 GMT T[AWT-EventQueue-0](26) DEBUG
> > > UserGroupInformation: UGI loginUser:whiro01 (auth:SIMPLE) Apr 11=2C20=
14
> > > 18:27:39.813 GMT T[AWT-EventQueue-0](26) ERROR HdfsFileSystem:
> > > Error connecting to filesystem hdfs://hadoop-master:50070/: No
> > > FileSystem for scheme: hdfs
> > > java.io.IOException: No FileSystem for scheme: hdfs
> > > at
> > > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:23=
04)
> > > at
> > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311=
)
> > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
> > > at
> > > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:235=
0)
> > > at
> > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:168)
> > > at
> > >
> > org.apache.commons.vfs2.provider.hdfs.HdfsFileSystem.resolveFile(HdfsFi=
leSystem.java:115)
> > > at
> > >
> > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFi=
le(AbstractOriginatingFileProvider.java:84)
> > > at
> > >
> > org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFi=
le(AbstractOriginatingFileProvider.java:64)
> > > at
> > >
> > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defau=
ltFileSystemManager.java:700)
> > > at
> > >
> > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Defau=
ltFileSystemManager.java:656)
> > > at
> > > org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(Def=
a
> > > ultFileSystemManager.java:609)
> > >
> > > So=2C my guess is that I don't have enough configuration setup on my
> > > client machine to tell Hadoop that the authentication is to be done a=
t
> > > the remote end ....?? So=2C I'm trying to track down what the
> > > configuration info might be.
> > >
> > > Just hoping there are some Hadoop users/experts here who might be abl=
e
> > > to help. Or maybe I need to talk to Hadoop Users list. Just thought
> > > since I was using Commons VFS that someone here could help (as in the
> > > author of the HdfsProvider ....)
> > >
> > > Note: I want to build a GUI component that can browse to arbitrary
> > > HDFS installations=2C so I can't really be setting up a hard-coded XM=
L
> > > file for each potential Hadoop cluster I might connect to ....
> > >
> > > Thanks=2C
> > > ~Roger
> > >
> > > -----Original Message-----
> > > From: Martin Gainty [mailto:mgainty@hotmail.com]
> > > Sent: Tuesday=2C April 08=2C 2014 5:56 PM
> > > To: Commons Users List
> > > Subject: RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS
> > > access?
> > >
> > > Roger-
> > >
> > >
> > >
> > > look for <artifactId>hadoop-core</artifactId>
> > > in the <dependencies><dependency> section in your pom.xml
> > >
> > > if either
> > >
> > > >mvn compile
> > >
> > > >mvn package
> > > cannot locate hadoop-core
> > > you'll need to sync your <distributionManagement> section in pom.xml
> > > for
> > >
> > > hadoop-core snapshots
> > > hadoop-core releases
> > >
> > >
> > > Ping back if you're calling no-joy
> > >
> > > M-
> > >
> > >
> > >
> > >
> > > > From: Roger.Whitcomb@actian.com
> > > > To: user@commons.apache.org
> > > > Subject: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS
> > access?
> > > > Date: Tue=2C 8 Apr 2014 21:46:09 +0000
> > > >
> > > > Hi all=2C
> > > > I've written a Commons VFS file browser using Apache Pivot (it's
> > > > part of
> > > the 2.1 upcoming release). Now I'm trying to use it to browse into
> > > HDFS files. I got it to work with HDFS 1.x by simple including the
> > > hadoop-core .jar and a couple of other commons .jars that are also
> > > needed. But=2C that approach doesn't work for Hadoop 2.x. Specificall=
y
> > > the "hadoop-core-xxx.jar" doesn't exist anymore=2C and substituting t=
he
> > > "hadoop-hdfs-xxxx.jar" doesn't work at all. So=2C before I bang my he=
ad
> > > against the wall=2C does anyone here know what .jars I would need for
> > > Hadoop 2.x to interface with the VFS HDFS provider?
> > > >
> > > > Thanks=2C
> > > > ~Roger Whitcomb
> > > >
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe=2C e-mail: user-unsubscribe@commons.apache.org
> > > For additional commands=2C e-mail: user-help@commons.apache.org
> > >
> > >
> >
> >
> > --
> > E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence
> > with Hibernate=2C Second Edition<http://www.manning.com/bauer3/>
> > JUnit in Action=2C Second Edition <http://www.manning.com/tahchiev/>
> > Spring Batch in Action <http://www.manning.com/templier/>
> > Blog: http://garygregory.wordpress.com
> > Home: http://garygregory.com/
> > Tweet! http://twitter.com/GaryGregory
> >
> > ---------------------------------------------------------------------
> > To unsubscribe=2C e-mail: user-unsubscribe@commons.apache.org
> > For additional commands=2C e-mail: user-help@commons.apache.org
> >
> >
>=20
>=20
> --=20
> E-Mail: garydgregory@gmail.com | ggregory@apache.org
> Java Persistence with Hibernate=2C Second Edition<http://www.manning.com/=
bauer3/>
> JUnit in Action=2C Second Edition <http://www.manning.com/tahchiev/>
> Spring Batch in Action <http://www.manning.com/templier/>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory
 		 	   		  =

--_15e7f6c3-a799-41be-a354-cf66a520f219_--