Mailing-List: contact user-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@giraph.apache.org
Received-SPF: pass (nike.apache.org: domain of alexandrejorgefonseca@gmail.com
 designates 209.85.214.53 as permitted sender)
From: Alexandre Fonseca <alexandrejorgefonseca@gmail.com>
To: user@giraph.apache.org
Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0)
Date: Mon, 09 Dec 2013 09:16:41 +0100
Message-ID: <1647021.5fbjhLDxPK@chronos7>
User-Agent: KMail/4.11.4 (Linux/3.12.3-1-ck; KDE/4.11.4; x86_64; ; )
In-Reply-To: 
 <CAFEw3Yi6XLCoPnNvH-KMazcjMwOBd0akztx4pUeG+61eRSeMww@mail.gmail.com>
References: <CEB0F400.2CDC4%rvesse@dotnetrdf.org>
 <CAOsFL5QdCVyiugkV4p-XyQgN4yf9=w+9cgUMTajKdNQggOy4WA@mail.gmail.com>
 <CAFEw3Yi6XLCoPnNvH-KMazcjMwOBd0akztx4pUeG+61eRSeMww@mail.gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="utf-8"

Hello Vishal,=20

Cross-posting from a different mailing list thread with the steps that =
allowed=20
me to compile and run SimpleShortestPaths on a pure-yarn cluster. Hope =
it=20
works for you too :)

I was able to compile and run Giraph 1.1.0-SNAPSHOT on top of hadoop-2.=
2.0.=20
Here's what I did:

        git clone https://git-wip-us.apache.org/repos/asf/giraph.git
        cd giraph
        git revert ac93c3b6c5bce5f22b293b29df91663ca7d7ce63
        mvn -Phadoop_yarn -Dhadoop.version=3D2.2.0 -DskipTests compile

With that I got valid jars which allowed me to run the SimpleShortestPa=
ths=20
example in my single-node setup Hadoop 2.2.0 + ZooKeeper 3.4.5:

        mkdir test_example
        ln -s $GIRAPH_PREFIX/giraph-examples/target/giraph-examples-1.1=
.0-
SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
        $HADOOP_PREFIX/bin/hadoop jar giraph-examples-1.1.0-SNAPSHOT-fo=
r-
hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner=20=

org.apache.giraph.examples.SimpleShortestPathsComputation -vif=20
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat=
 -vip=20
giraph/input/tiny_graph.txt -vof=20
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op=20
giraph/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=3Dfalse -=
ca=20
giraph.zkList=3Dlocalhost:2181 -yj giraph-examples-1.1.0-SNAPSHOT-for-
hadoop-2.2.0-jar-with-dependencies.jar

###########
Notes:
* Wasn't able to do mvn package as it complained about not having the=20=

dependencies giraph-hive, giraph-hbase, etc... I suspect this is just a=
 matter=20
of changing the main pom.xml to include those modules with the hadoop 2=
.2.0=20
target or remove those dependencies from the giraph-dist if they no lon=
ger=20
apply.
* You may get some memory limit exceeded problems for the ApplicationMa=
ster.=20
To fix, either increase the yarn.nodemanager.vmem-pmem-ratio parameter =
in yarn-
site.xml or apply my patch to prevent allocation of heap size equal to =
total=20
memory size, described at: https://issues.apache.org/jira/browse/GIRAPH=
-812
* In my case, the application didn't exit even after outputting the res=
ult and
having recognized that all nodes voted to terminate. I've opened an iss=
ue
in JIRA regarding this so if you suffer from the same, you can apply th=
at patch
too: https://issues.apache.org/jira/browse/GIRAPH-811

###########
Relevant JIRA issues:
* https://issues.apache.org/jira/browse/GIRAPH-798
* https://issues.apache.org/jira/browse/GIRAPH-804
* https://issues.apache.org/jira/browse/GIRAPH-811
* https://issues.apache.org/jira/browse/GIRAPH-812

Best regards,
Alexandre

On Sunday 08 December 2013 14:52:15 Vishal Mishra wrote:
> Setup: hadoop 2.2.0 on a 2-node cluster on ubuntu 13.10.
> Giraph: sudo git clone https://github.com/apache/giraph.git
>=20
> Tried a few things in the past but was not able to get even one succe=
ssful
> run for the basic example. Our cluster is healthy and we've been runn=
ing MR
> and Hive successfully for over 3 weeks.
>=20
> Just started from scratch again and now I can't get past the build fo=
r the
> latest as several dependency checks are failing:
>=20
> [ERROR] Found a problem with the dependency
> commons-httpclient:commons-httpclient
>   Resolved version is 3.0.1
>   Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-comm=
on
> [ERROR] Found a problem with the dependency commons-logging:commons-l=
ogging
>   Resolved version is 1.0.4
>   Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-co=
mmon
> [ERROR] Found a problem with the dependency io.netty:netty
>   Resolved version is 3.5.3.Final
>   Version 3.6.2.Final was expected by artifacts:
> org.apache.hadoop:hadoop-mapreduce-client-common,
> org.apache.hadoop:hadoop-mapreduce-client-core,
> org.apache.hadoop:hadoop-yarn-common,
> org.apache.hadoop:hadoop-yarn-server-common,
> org.apache.hadoop:hadoop-yarn-server-nodemanager,
> org.apache.hadoop:hadoop-yarn-server-resourcemanager,
> org.apache.hadoop:hadoop-yarn-server-tests:test-jar
> [ERROR] Found a problem with the dependency org.apache.zookeeper:zook=
eeper
>   Resolved version is 3.3.3
>   Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-c=
ommon,
> org.apache.hadoop:hadoop-mapreduce-client-common,
> org.apache.hadoop:hadoop-yarn-server-common,
> org.apache.hadoop:hadoop-yarn-server-nodemanager,
> org.apache.hadoop:hadoop-yarn-server-resourcemanager,
> org.apache.hadoop:hadoop-yarn-server-tests:test-jar
> the building is failing due to dependency check issues
>=20
>=20
> Would really appreciate it if someone can share the correct branch an=
d
> steps that have been successfully applied in a cluster. I'd be more t=
han
> happy to document the steps and post them for everyone.
>=20
> Thanx,
> Vishal
>=20
> On Sun, Dec 8, 2013 at 10:23 AM, Eli Reisman=20
<apache.mailbox@gmail.com>wrote:
> > Hi all,
> >=20
> > It looks to me like you want to run the giraph-core jar at the begi=
nning
> > of the "hadoop jar" call and supply the giraph-examples jar with -y=
j
> > argument.
> >=20
> > Also, you can place any -D type configuration settings you're sick =
of
> > supplying at the command line into giraph-site.xml just as you woul=
d with
> > Hadoop settings in one of its own site.xml files, and place these w=
here
> > your local Hadoop client configs are and they should be picked up i=
n your
> > application.
> >=20
> > Hope that helps,
> >=20
> > Eli
> >=20
> > On Thu, Nov 28, 2013 at 12:21 PM, Vishal Mishra=20
<mishravishal@gmail.com>wrote:
> >> I had a mistake in my command (moronic copy/paste zookeeper settin=
gs
> >> error).
> >> Updated and now hitting the following:
> >>=20
> >> ....
> >> 13/11/28 12:02:14 INFO yarn.GiraphYarnClient: Running Client
> >> 13/11/28 12:02:14 INFO client.RMProxy: Connecting to ResourceManag=
er at
> >> master/192.168.1.100:8045
> >> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from=
 ASM
> >> for, nodeId=3Dkreator:46477, nodeAddresskreator:8042,
> >> nodeRackName/default-rack, nodeNumContainers7
> >> 13/11/28 12:02:15 INFO yarn.GiraphYarnClient: Got node report from=
 ASM
> >> for, nodeId=3Dexotica:46645, nodeAddressexotica:8042,
> >> nodeRackName/default-rack, nodeNumContainers8
> >> Exception in thread "main" java.lang.RuntimeException: Giraph job
> >> requires 2 containers to run; cluster only hosts 15
> >>=20
> >>  at
> >>=20
> >> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvail=
able(Gi
> >> raphYarnClient.java:230) at
> >> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:=
125)
> >> ....
> >>=20
> >> It seems like some config is missing from our hadoop setup.
> >> Also, note that I'm working from the trunk branch.
> >>=20
> >> Thanx,
> >> Vishal
> >>=20
> >> On Wed, Nov 27, 2013 at 11:34 AM, Vishal Mishra=20
<mishravishal@gmail.com>wrote:
> >>> Hey Guys,
> >>> I'm trying giraph 1.0.0 on hadoop 2.2.0 (2-node cluster) but hitt=
ing the
> >>> same problem as outlined in this thread. Was there a resolution? =
If yes,
> >>> can you please share?
> >>> Completely new to giraph and hadoop so please excuse any omission=
s. Our
> >>> cluster is fully functional and running mapreduce and hive jobs c=
leanly.
> >>>=20
> >>> Command:
> >>> hadoop jar
> >>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHO=
T-for-h
> >>> adoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRun=
ner -D
> >>> giraph.zkList=3D"zkNode.net:2081"
> >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInput=
Format
> >>> -vip /user/hduser/rrdata/tiny_graph.txt -vof
> >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>> /user/hduser/rrdata/output -w 1 -yj
> >>> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHO=
T-for-h
> >>> adoop-2.2.0-jar-with-dependencies.jar
> >>>=20
> >>> Trimmed Console Output:
> >>> 13/11/27 11:19:58 INFO yarn.GiraphYarnClient:
> >>> appattempt_1385518875931_0002_000001, State: ACCEPTED, Containers=
 used:
> >>> 1
> >>> 13/11/27 11:20:00 ERROR yarn.GiraphYarnClient: Giraph:
> >>> org.apache.giraph.examples.SimpleShortestPathsComputation reports=
 FAILED
> >>> state, diagnostics show: Application application_1385518875931_00=
02
> >>> failed
> >>> 2 times due to AM Container for appattempt_1385518875931_0002_000=
002
> >>> exited
> >>>=20
> >>> with  exitCode: 1 due to: Exception from container-launch:
> >>>  org.apache.hadoop.util.Shell$ExitCodeException:
> >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> >>> at org.apache.hadoop.util.Shell.run(Shell.java:379)
> >>>=20
> >>>  at
> >>>=20
> >>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.j=
ava:589
> >>> )
> >>> at
> >>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecuto=
r.launc
> >>> hContainer(DefaultContainerExecutor.java:195)>>>=20
> >>>  at
> >>>=20
> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launch=
er.Cont
> >>> ainerLaunch.call(ContainerLaunch.java:283) at
> >>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launch=
er.Cont
> >>> ainerLaunch.call(ContainerLaunch.java:79)>>>=20
> >>>  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>=20
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecu=
tor.jav
> >>> a:1145)>>>=20
> >>>  at
> >>>=20
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec=
utor.ja
> >>> va:615) at java.lang.Thread.run(Thread.java:744)
> >>>=20
> >>>=20
> >>> Thanx,
> >>> Vishal
> >>>=20
> >>> On Tue, Nov 19, 2013 at 3:16 PM, Mohammad Islam=20
<mislam77@yahoo.com>wrote:
> >>>> Rob,
> >>>> I understand your frustration.
> >>>> I'm pre-occupied with some other stuffs in next couple of days.
> >>>> If no one replied by then, I will recreate the situation here an=
d try
> >>>> to give you more concrete help.
> >>>>=20
> >>>> Regards,
> >>>> Mohammad
> >>>>=20
> >>>>   On Tuesday, November 19, 2013 2:56 AM, Rob Vesse <
> >>>>=20
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>>   Mohammad
> >>>>=20
> >>>> Comments inline:
> >>>>=20
> >>>> From: Mohammad Islam <mislam77@yahoo.com>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Monday, 18 November 2013 22:24
> >>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
> >>>> Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and =
Hadoop
> >>>> 2.2.0)
> >>>>=20
> >>>> I used the following shell script to run PR where my current wor=
king
> >>>> directory contains the graph*.jar in Hadoop 2.1.1.
> >>>> . You can try something similar.
> >>>>=20
> >>>>=20
> >>>> I already am.
> >>>>=20
> >>>>=20
> >>>>=20
> >>>>=20
> >>>> HAD_VERSION=3D2.1.1-SNAPSHOT
> >>>> hadoop fs -rmr out
> >>>> export GIRAPH_HOME=3D/home/mislam/softwares/giraph
> >>>> #export
> >>>> GIRAPH_CORE=3D$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPS=
HOT-for-h
> >>>> adoop-$HAD_VERSION-jar-with-dependencies.jar #export
> >>>> GIRAPH_EXAMPLE=3D$GIRAPH_HOME/giraph-examples/target/giraph-exam=
ples-1.1.
> >>>> 0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
> >>>>=20
> >>>>=20
> >>>> CLASSPATH=3D$CLASSPATH:$GIRAPH_HOME/conf
> >>>>=20
> >>>>=20
> >>>> What is the point of putting the conf directory on the CLASSPATH=
?  I
> >>>> assume this is only to pick up the giraph configuration but at t=
he same
> >>>> time there is nothing to stop me putting my Giraph specific conf=
ig in
> >>>> one
> >>>> of the other Hadoop configuration files e.g. core-site.xml?
> >>>>=20
> >>>> #CLASSPATH=3D$CLASSPATH:$GIRAPH_CORE
> >>>> #CLASSPATH=3D$CLASSPATH:$GIRAPH_EXAMPLE
> >>>>=20
> >>>> export CLASSPATH=3D$CLASSPATH
> >>>> export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$CLASSPATH
> >>>>=20
> >>>> echo $HADOOP_CLASSPATH
> >>>>=20
> >>>>=20
> >>>> I assume the point of the above is to ensure that the Giraph con=
fig
> >>>> file is on the class path Hadoop actually uses but adding this m=
akes no
> >>>> difference to me, I get exactly the same failure as before.  Mod=
ifying
> >>>> the
> >>>> logic to add the examples JAR onto the class path does not help =
either.
> >>>>=20
> >>>>=20
> >>>>=20
> >>>> hadoop --config $HADOOP_HOME/etc/hadoop jar
> >>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-=
depende
> >>>> ncies.jar>>>>=20
> >>>>  org.apache.giraph.GiraphRunner
> >>>>=20
> >>>> org.apache.giraph.examples.SimplePageRankComputation -c
> >>>> org.apache.giraph.combiner.DoubleSumCombiner -w 1 -yh 1024 -yj
> >>>> giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-=
depende
> >>>> ncies.jar -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -m=
c
> >>>> org.apache.giraph.examples.SimplePageRankComputation\$SimplePage=
RankMas
> >>>> terCompute -vip input -op out
> >>>>=20
> >>>>=20
> >>>> This is not the same command that I am running, also DoubleSumCo=
mbiner
> >>>> gives a ClassNotFoundException, you actually want
> >>>> DoubleSumMessageCombiner.
> >>>>=20
> >>>>  If I adapt my script to run your command I experience the exact=
 same
> >>>>  error
> >>>>=20
> >>>> as I do trying to run the command I showed so I am still nowhere=
.
> >>>>=20
> >>>> I'm getting a little frustrated that no-one has any idea what th=
e
> >>>> problem is here, either this is a simple misconfiguration on my =
part or
> >>>> a
> >>>> bug in Giraph for Hadoop 2.2.0 ?  Given the thread so far and th=
e fact
> >>>> that
> >>>> I can run MR based Giraph on Hadoop 1.2.1 this is looking increa=
singly
> >>>> unlikely I'm starting to lean heavily towards the latter option.=

> >>>>=20
> >>>> Rob
> >>>>=20
> >>>>   On Monday, November 18, 2013 2:24 AM, Rob Vesse
> >>>>   <rvesse@dotnetrdf.org>
> >>>>=20
> >>>> wrote:
> >>>>  Hey all
> >>>>=20
> >>>> Bumping this thread because I am still completely blocked on thi=
s and
> >>>> hoping someone can help me.  As explained earlier in the thread =
I have
> >>>> Giraph built for pure YARN on stable Hadoop 2.2.0 release like s=
o:
> >>>>=20
> >>>> mvn =E2=80=93Phadoop_yarn =E2=80=93Dhadoop.version=3D2.2.0 =E2=80=
=93DskipTests=3Dtrue clean install
> >>>>=20
> >>>> But when I try and run the simple shortest paths demo like so:
> >>>>=20
> >>>> $> bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths -w 1 -yj
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>>=20
> >>>> The application fails and the YARN logs show the following error=
:
> >>>>=20
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> org/apache/giraph/yarn/GiraphApplicationMaster Caused by:
> >>>> java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>=20
> >>>> =09at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> =09at java.security.AccessController.doPrivileged(Native Method)=

> >>>> =09at java.net.URLClassLoader.findClass(URLClassLoader.java:190)=

> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> =09at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:3=
01)
> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>=20
> >>>> Any ideas why the GiraphApplicationMaster would not be found?  H=
ave I
> >>>> just got something basic wrong in my configuration or is this a =
bug in
> >>>> Giraph?
> >>>>=20
> >>>>=20
> >>>> I found an earlier thread on the same topic
> >>>> (http://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbo=
x/%3c5
> >>>> 1E5A4D1.40006@googlemail.com%3e) but it was never resolved hence=
 why I
> >>>> have now changed the topic in the hope it will jog someone's mem=
ory
> >>>>=20
> >>>>=20
> >>>> Cheers,
> >>>>=20
> >>>>=20
> >>>> Rob
> >>>>=20
> >>>>=20
> >>>> From: Rob Vesse <rvesse@dotnetrdf.org>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Friday, 15 November 2013 14:14
> >>>> To: <user@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>=20
> >>>> Adding the =E2=80=93yj argument to specify the YARN JARs doesn't=
 make any
> >>>> difference.
> >>>>=20
> >>>> Here is the complete console output from my run:
> >>>>=20
> >>>> bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/3 -w 1 -yj
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>> 2013-11-15 14:11:22.897 java[4877:1203] Unable to load realm inf=
o from
> >>>> SCDynamicStore
> >>>> 13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load
> >>>> native-hadoop library for your platform... using builtin-java cl=
asses
> >>>> where
> >>>> applicable
> >>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge input f=
ormat
> >>>> specified. Ensure your InputFormat does not require one.
> >>>> 13/11/15 14:11:24 INFO utils.ConfigurationUtils: No edge output =
format
> >>>> specified. Ensure your OutputFormat does not require one.
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path =
is:
> >>>> hdfs://localhost/user/rvesse/shortestpaths/3
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Client
> >>>> 13/11/15 14:11:24 INFO client.RMProxy: Connecting to ResourceMan=
ager at
> >>>> /0.0.0.0:8032
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got node report fr=
om ASM
> >>>> for, nodeId=3Dmbp-rvesse.home:49981, nodeAddressmbp-rvesse.home:=
8042,
> >>>> nodeRackName/default-rack, nodeNumContainers0
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-tas=
k heap
> >>>> size to 1024MB.
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Appli=
cation
> >>>> ID: application_1384523910015_0004
> >>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id =
is
> >>>> deprecated. Instead, use mapreduce.job.id
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environmen=
t for
> >>>> the application master
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Environment for AM=

> >>>>=20
> >>>> :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_H=
OME/share
> >>>> :/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:=
$HADOOP
> >>>> :_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/h=
dfs/lib
> >>>> :/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/shar=
e/hadoo
> >>>> :p/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADO=
OP_MAPR
> >>>> :ED_HOME/share/hadoop/mapreduce/lib/*}>>>>=20
> >>>> 13/11/15 14:11:24 INFO yarn.GiraphYarnClient: buildLocalResource=
Map
> >>>> ....
> >>>> 13/11/15 14:11:24 INFO Configuration.deprecation: mapred.output.=
dir is
> >>>> deprecated. Instead, use mapreduce.output.fileoutputformat.outpu=
tdir
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Registered file in
> >>>> LocalResources ::
> >>>> hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_1=
3845239
> >>>> 10015_0004/giraph-conf.xml 13/11/15 14:11:25 INFO
> >>>> yarn.GiraphYarnClient: LIB JARS
> >>>>=20
> >>>> :/Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target=
/giraph
> >>>> :-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-
dependencies.jar>>>>=20
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking .
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name null
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/li=
b/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/li=
b
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/=
*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/=
*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce=
/lib/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce=
/lib
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce=
/*
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce=

> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name
> >>>> /contrib/capacity-scheduler/*.jar
> >>>> 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking
> >>>> /contrib/capacity-scheduler
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient:
> >>>> ApplicationSumbissionContext for GiraphApplicationMaster launch
> >>>> container
> >>>> is populated.
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting applica=
tion to
> >>>> ASM
> >>>> 13/11/15 14:11:25 INFO impl.YarnClientImpl: Submitted applicatio=
n
> >>>> application_1384523910015_0004 to ResourceManager at /0.0.0.0:80=
32
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId afte=
r
> >>>> submission :application_1384523910015_0004
> >>>> 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphApplicationM=
aster
> >>>> container request was submitted to ResourceManager for job: Gira=
ph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation
> >>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient: Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elaps=
ed:
> >>>> 0.82
> >>>> secs
> >>>> 13/11/15 14:11:26 INFO yarn.GiraphYarnClient:
> >>>> appattempt_1384523910015_0004_000001, State: ACCEPTED, Container=
s used:
> >>>> 1
> >>>> 13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation report=
s
> >>>> FAILED
> >>>> state, diagnostics show: Application application_1384523910015_0=
004
> >>>> failed
> >>>> 2 times due to AM Container for appattempt_1384523910015_0004_00=
0002
> >>>> exited
> >>>> with  exitCode: 1 due to: Exception from container-launch:
> >>>> org.apache.hadoop.util.Shell$ExitCodeException:
> >>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
> >>>>=20
> >>>>  at org.apache.hadoop.util.Shell.run(Shell.java:379)
> >>>>=20
> >>>> at
> >>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.=
java:58
> >>>> 9)
> >>>>=20
> >>>>  at
> >>>>=20
> >>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecut=
or.laun
> >>>> chContainer(DefaultContainerExecutor.java:195) at
> >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launc=
her.Con
> >>>> tainerLaunch.call(ContainerLaunch.java:283) at
> >>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launc=
her.Con
> >>>> tainerLaunch.call(ContainerLaunch.java:79)>>>>=20
> >>>>  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.jav=
a:303)
> >>>>=20
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>=20
> >>>>  at
> >>>>=20
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo=
lExecut
> >>>> or.java:895) at
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe=
cutor.j
> >>>> ava:918)>>>>=20
> >>>>  at java.lang.Thread.run(Thread.java:695)
> >>>>=20
> >>>> .Failing this attempt.. Failing the application.
> >>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS
> >>>> distributed cache directory for Giraph job.
> >>>> 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Completed Giraph:
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation: FAILE=
D,
> >>>> total
> >>>> running time: 0 minutes, 3 seconds.
> >>>>=20
> >>>> And I see the same error in the logs:
> >>>>=20
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>> org/apache/giraph/yarn/GiraphApplicationMaster Caused by:
> >>>> java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>=20
> >>>> =09at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> =09at java.security.AccessController.doPrivileged(Native Method)=

> >>>> =09at java.net.URLClassLoader.findClass(URLClassLoader.java:190)=

> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> =09at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:3=
01)
> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>=20
> >>>> Any more ideas?
> >>>>=20
> >>>>=20
> >>>>=20
> >>>>=20
> >>>> Rob
> >>>>=20
> >>>>=20
> >>>> From: Mohammad Islam <mislam77@yahoo.com>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Thursday, 14 November 2013 20:36
> >>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>=20
> >>>> I tried the following command few months back. I found you are n=
ot
> >>>> passing some params.
> >>>>=20
> >>>> bin/hadoop --config etc/hadoop jar
> >>>> /export/home/clei/giraph/giraph-1.0.0/giraph-examples/target/gir=
aph-exa
> >>>> mples-1.?.0-for-hadoop-???-jar-with-dependencies.jar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimplePageRankVertexOpt -c
> >>>> org.apache.giraph.combiner.DoubleSumCombiner -w 35 -yh 20480 -yj=

> >>>> giraph-examples-1.?.?-for-hadoop-2.??-jar-with-dependencies.jar =
-vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -of
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -vip
> >>>> /user/me/input -op /user/me/output_35w_20G_80m_1
> >>>>=20
> >>>>   On Thursday, November 14, 2013 3:22 AM, Rob Vesse <
> >>>>=20
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>>  It's the same command line I showed earlier, just trying to run=
 the
> >>>>=20
> >>>> example application:
> >>>>=20
> >>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/2 -w 1
> >>>>=20
> >>>> Rob
> >>>>=20
> >>>> From: Mohammad Islam <mislam77@yahoo.com>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Thursday, 14 November 2013 05:57
> >>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>=20
> >>>> What are you running? PR?
> >>>> What is the command line?
> >>>>=20
> >>>>   On Wednesday, November 13, 2013 8:30 AM, Rob Vesse <
> >>>>=20
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>>  So I fixed the previous issue I was having which I reported and=

> >>>>=20
> >>>> submitted a patch as GIRAPH-795
> >>>>=20
> >>>> I then went ahead with installing and configuring my external Zo=
okeeper
> >>>> appropriately and tried running again but now the YARN applicati=
on
> >>>> fails
> >>>> and I find the following error in the logs:
> >>>>=20
> >>>> Exception in thread "main" java.lang.NoClassDefFoundError:
> >>>>  org/apache/giraph/yarn/GiraphApplicationMaster
> >>>>=20
> >>>> Caused by: java.lang.ClassNotFoundException:
> >>>> org.apache.giraph.yarn.GiraphApplicationMaster>>>>=20
> >>>> =09at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >>>> =09at java.security.AccessController.doPrivileged(Native Method)=

> >>>> =09at java.net.URLClassLoader.findClass(URLClassLoader.java:190)=

> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> >>>> =09at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:3=
01)
> >>>> =09at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> >>>>=20
> >>>> Anybody have idea what the cause of this one might be or how to =
fix it?
> >>>>=20
> >>>>=20
> >>>> Cheers,
> >>>>=20
> >>>>=20
> >>>> Rob
> >>>>=20
> >>>>=20
> >>>> From: Rob Vesse <rvesse@dotnetrdf.org>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Wednesday, 13 November 2013 15:35
> >>>> To: <user@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>=20
> >>>> Mohammad
> >>>>=20
> >>>> This got me a build that at least attempts to start to run but n=
ot I
> >>>> see the following:
> >>>>=20
> >>>> mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar
> >>>> /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/=
giraph-
> >>>> examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.j=
ar
> >>>> org.apache.giraph.GiraphRunner
> >>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> >>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu=
tFormat
> >>>> -vip /user/rvesse/tiny_graph.txt -vof
> >>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> >>>> /user/rvesse/shortestpaths/2 -w 1
> >>>> 2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm in=
fo from
> >>>> SCDynamicStore
> >>>> 13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable to load
> >>>> native-hadoop library for your platform... using builtin-java cl=
asses
> >>>> where
> >>>> applicable
> >>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge input f=
ormat
> >>>> specified. Ensure your InputFormat does not require one.
> >>>> 13/11/13 15:13:31 INFO utils.ConfigurationUtils: No edge output =
format
> >>>> specified. Ensure your OutputFormat does not require one.
> >>>> 13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path =
is:
> >>>> hdfs://localhost/user/rvesse/shortestpaths/2
> >>>> Exception in thread "main" java.lang.NullPointerException
> >>>>=20
> >>>>  at
> >>>>=20
> >>>> org.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSu=
pported
> >>>> (GiraphYarnClient.java:460) at
> >>>> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.jav=
a:116)
> >>>>=20
> >>>>  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
> >>>>=20
> >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >>>>=20
> >>>>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >>>>=20
> >>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
> >>>>=20
> >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>=20
> >>>> at
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor=
Impl.ja
> >>>> va:39)>>>>=20
> >>>>  at
> >>>>=20
> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
Accesso
> >>>> rImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:59=
7)
> >>>>=20
> >>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >>>>=20
> >>>> Which indirectly implies that I have some missing configuration =
to run
> >>>> Giraph on YARN.  I don't think that should be a NPE and it looks=
 to be
> >>>> a
> >>>> trivial fix so I'll try and fix and send a pull request for that=
.
> >>>>=20
> >>>> The error message I should receive if the NPE wasn't happening i=
s the
> >>>> following:
> >>>>=20
> >>>> Giraph on YARN does not currentlysupport Giraph-managed ZK insta=
nces:
> >>>> use a standalone ZooKeeper.
> >>>>=20
> >>>> What configuration options do I need to have set to point Giraph=
 at an
> >>>> external Zookeeper?
> >>>>=20
> >>>> Cheers,
> >>>>=20
> >>>> Rob
> >>>>=20
> >>>> From: Mohammad Islam <mislam77@yahoo.com>
> >>>> Reply-To: <user@giraph.apache.org>
> >>>> Date: Wednesday, 13 November 2013 11:15
> >>>> To: "user@giraph.apache.org" <user@giraph.apache.org>
> >>>> Subject: Re: Giraph and Hadoop 2.2.0
> >>>>=20
> >>>> Hi Rob,
> >>>> Did you try with this command "mvn -Phadoop_yarn -Dhadoop.versio=
n=3D2.2.0
> >>>> clean package -DskipTests"?
> >>>> Regards,
> >>>> Mohammad
> >>>>=20
> >>>>   On Wednesday, November 13, 2013 2:55 AM, Rob Vesse <
> >>>>=20
> >>>> rvesse@dotnetrdf.org> wrote:
> >>>>  Hi all
> >>>>=20
> >>>> I'm having trouble getting Giraph to build and run against a Had=
oop
> >>>> 2.2.0 installation.  I tried the hadoop_2.0.3 profile with no lu=
ck
> >>>> because
> >>>> I get class cast errors when trying to run the examples.
> >>>>=20
> >>>> Is there a specific profile I should be building to support Hado=
op
> >>>> 2.2.0 or is this not supported yet?
> >>>>=20
> >>>> Thanks,
> >>>>=20
> >>>> Rob