Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4AA9A10268 for ; Mon, 18 Nov 2013 22:24:44 +0000 (UTC) Received: (qmail 69776 invoked by uid 500); 18 Nov 2013 22:24:44 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 69744 invoked by uid 500); 18 Nov 2013 22:24:44 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 69733 invoked by uid 99); 18 Nov 2013 22:24:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Nov 2013 22:24:44 +0000 X-ASF-Spam-Status: No, hits=4.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLYTO_END_DIGIT,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.138.91.108] (HELO nm15-vm6.bullet.mail.ne1.yahoo.com) (98.138.91.108) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Nov 2013 22:24:37 +0000 Received: from [98.138.226.176] by nm15.bullet.mail.ne1.yahoo.com with NNFMP; 18 Nov 2013 22:24:16 -0000 Received: from [98.138.88.237] by tm11.bullet.mail.ne1.yahoo.com with NNFMP; 18 Nov 2013 22:24:16 -0000 Received: from [127.0.0.1] by omp1037.mail.ne1.yahoo.com with NNFMP; 18 Nov 2013 22:24:16 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 830361.46088.bm@omp1037.mail.ne1.yahoo.com Received: (qmail 5067 invoked by uid 60001); 18 Nov 2013 22:24:16 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1384813456; bh=beBkftVTWQmLS8+f5IF2OoUDrSizhzkW6fMxDVMvtqc=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=UnE6qIcfVImRu/IacnHCJoUKO1y9Jj8ovqY9cr0nUD9ndUHXcFipOoihjHU9jMn5Ilt/WsZsCxot676PNIe8c1dalT6qX1RhaRyhftiKXgrac23eN5SHsknClGGzKN+criU1327UZi/B2K3/Kw+TvPdopT0Q9DF021b3qfq2EIQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=itMF/M3C7gnZ8st58L7RfyiDYvMuZB1SlWjpelDpzsXqefMmAIOA9Fj+KwTbb7bc0UX/8QG49plp5AStbPBELL9kfYf/60bnZ+YpQ8l8r318b6Go0hEN6bt2f+CmH1CkaNXXZLx4iY2P5bMuEcKAB9+BNi3traU8MoG9JX0ow7c=; X-YMail-OSG: JjBbDwQVM1n3QrEK5q2iGKBCE1w97tO7vVK4HWhSeKX0.22 3v4VHc5WkSSQCqGkFD_Y8SDYldKDnw9Y8diSCgZxcusvfl9z6LAXh1rW7zE0 IMOWXvLFGKIew.NFeVrczHJhoCaU83QjT0qSDsaazdUPOgQTd1kM5iZYU.9A bv4_PTG4kfxX.f.LK1mn.By7dFaL3gZFbR6Q8mnaLJBWW5Xf0JwQ.iIvMqSW MzIL5zzbw_5DFJTlOS4rYo8e.pKn3S7XIhTh7v0xcEb6rNpFmtduE5vQAmEb w4QW.vJjJ7Q2l5mnAMOuCouyGaxNANZbnE0vRTQkRCFs2M8ExmDAkOMM5O10 2ij43TEAnxmQgDqIQAQCtYbLdfpwLDuHjZOc6cJo7IHP6IiCm6E2c5OHiP92 wTQn_AVpQ.3BjWbsvkTkpHh2ixFvWP7NVAMjV3xZcpEPvQ37UW8fC_.0RLZo 5UzgnnwPYs.Dh8qAM8yP69tebnyTevoDn6_0GL_0bAkm4bNMRtnNJkLi6onm rMcN.k7Tim_NdfRcT.UPtm1IDPli.NdiGiJ4Oi.hL0ukBmAvuU9CSfxnQ2Sc rUY.azZxfHS0g1G8bfqyMuktBW8WbtbLEsiQrw8.q5ALMQfntLsKlJ26Mbf3 tZ23NCB_a7dIpDRm5o16uOLU59yeBQd1cYTz0IuTuamQ7ig3RiYOG_84Lwxb 5HFwMeLgHQfYtsBceORsyd6Ep67X0g2isQGluFmPJY_MuUCJDLXd4HcjxyF1 avZCW8lsGCwRGps3ollebqCnFxw7krK_d4cP.fw-- Received: from [69.28.149.129] by web121701.mail.ne1.yahoo.com via HTTP; Mon, 18 Nov 2013 14:24:16 PST X-Rocket-MIMEInfo: 002.001,SSB1c2VkIHRoZSBmb2xsb3dpbmcgc2hlbGwgc2NyaXB0IHRvIHJ1biBQUiB3aGVyZSBteSBjdXJyZW50IHdvcmtpbmcgZGlyZWN0b3J5IGNvbnRhaW5zIHRoZSBncmFwaCouamFyIGluIEhhZG9vcCAyLjEuMS4KLiBZb3UgY2FuIHRyeSBzb21ldGhpbmcgc2ltaWxhci4KCgoKSEFEX1ZFUlNJT049Mi4xLjEtU05BUFNIT1QKaGFkb29wIGZzIC1ybXIgb3V0CmV4cG9ydCBHSVJBUEhfSE9NRT0vaG9tZS9taXNsYW0vc29mdHdhcmVzL2dpcmFwaAojZXhwb3J0IEdJUkFQSF9DT1JFPSRHSVJBUEhfSE9NRS9naXJhcGgBMAEBAQE- X-Mailer: YahooMailWebService/0.8.166.601 References: Message-ID: <1384813456.4648.YahooMailNeo@web121701.mail.ne1.yahoo.com> Date: Mon, 18 Nov 2013 14:24:16 -0800 (PST) From: Mohammad Islam Reply-To: Mohammad Islam Subject: Re: GiraphApplicationMaster not found (WAS: Giraph and Hadoop 2.2.0) To: "user@giraph.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-887950874-152249639-1384813456=:4648" X-Virus-Checked: Checked by ClamAV on apache.org ---887950874-152249639-1384813456=:4648 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I used the following shell script to run PR where my current working direct= ory contains the graph*.jar in Hadoop 2.1.1.=0A. You can try something simi= lar.=0A=0A=0A=0AHAD_VERSION=3D2.1.1-SNAPSHOT=0Ahadoop fs -rmr out=0Aexport = GIRAPH_HOME=3D/home/mislam/softwares/giraph=0A#export GIRAPH_CORE=3D$GIRAPH= _HOME/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-= with-dependencies.jar=0A#export GIRAPH_EXAMPLE=3D$GIRAPH_HOME/giraph-exampl= es/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-with-d= ependencies.jar=0A=0A=0ACLASSPATH=3D$CLASSPATH:$GIRAPH_HOME/conf=0A#CLASSPA= TH=3D$CLASSPATH:$GIRAPH_CORE=0A#CLASSPATH=3D$CLASSPATH:$GIRAPH_EXAMPLE=0A= =0Aexport CLASSPATH=3D$CLASSPATH=0Aexport HADOOP_CLASSPATH=3D$HADOOP_CLASSP= ATH:$CLASSPATH=0A=0Aecho $HADOOP_CLASSPATH=0A=0A=0Ahadoop --config $HADOOP_= HOME/etc/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-= jar-with-dependencies.jar =C2=A0org.apache.giraph.GiraphRunner org.apache.g= iraph.examples.SimplePageRankComputation -c org.apache.giraph.combiner.Doub= leSumCombiner -w 1 -yh 1024 -yj giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$= HAD_VERSION-jar-with-dependencies.jar -vif org.apache.giraph.io.formats.Jso= nLongDoubleFloatDoubleVertexInputFormat -vof org.apache.giraph.io.formats.I= dWithValueTextOutputFormat -mc org.apache.giraph.examples.SimplePageRankCom= putation\$SimplePageRankMasterCompute -vip input -op out=0A=0A=0A=0AOn Mond= ay, November 18, 2013 2:24 AM, Rob Vesse wrote:=0A = =0AHey all=0A=0ABumping this thread because I am still completely blocked o= n this and hoping someone can help me. =C2=A0As explained earlier in the th= read I have Giraph built for pure YARN on stable Hadoop 2.2.0 release like = so:=0A=0Amvn =E2=80=93Phadoop_yarn =E2=80=93Dhadoop.version=3D2.2.0 =E2=80= =93DskipTests=3Dtrue clean install=0A=0ABut when I try and run the simple s= hortest paths demo like so:=0A=0A$> bin/hadoop jar /Users/rvesse/Documents/= Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-= hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.a= pache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph= .io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/ti= ny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat = -op /user/rvesse/shortestpaths -w 1 -yj /Users/rvesse/Documents/Work/Code/g= iraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.= 0-jar-with-dependencies.jar=0A=0AThe application fails and the YARN logs sh= ow the following error:=0A=0AException in thread "main" java.lang.NoClassDe= fFoundError: org/apache/giraph/yarn/GiraphApplicationMaster=0ACaused by: ja= va.lang.ClassNotFoundException: org.apache.giraph.yarn.GiraphApplicationMas= ter at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.secur= ity.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader= .findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(Clas= sLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.ja= va:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247)=0A=0AAny i= deas why the GiraphApplicationMaster would not be found? Have Ijust got so= mething basic wrong in my configuration or is this a bug in Giraph?=0A=0AIf= ound an earlier thread on the same topic (http://mail-archives.apache.org/m= od_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40006@googlemail.com%3e)but it = was never resolved hence why Ihave now changed the topic in the hope it wil= l jog someone's memory=0A=0ACheers,=0A=0ARob=0A=0AFrom: Rob Vesse =0AReply-To: =0ADate: Friday, 15 No= vember 2013 14:14=0ATo: =0ASubject: Re: Giraph an= d Hadoop 2.2.0=0A=0A=0AAdding the =E2=80=93yj argument to specify the YARN = JARs doesn't make any difference.=0A>=0A>=0A>Here is the complete console o= utput from my run:=0A>=0A>=0A>bin/hadoop jar /Users/rvesse/Documents/Work/C= ode/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= -2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.= giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.fo= rmats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_gra= ph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /u= ser/rvesse/shortestpaths/3 -w 1 -yj /Users/rvesse/Documents/Work/Code/girap= h/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-ja= r-with-dependencies.jar=0A>2013-11-15 14:11:22.897 java[4877:1203] Unable t= o load realm info from SCDynamicStore=0A>13/11/15 14:11:22 WARN util.Native= CodeLoader: Unable to load native-hadoop library for your platform... using= builtin-java classes where applicable=0A>13/11/15 14:11:24 INFO utils.Conf= igurationUtils: No edge input format specified. Ensure your InputFormat doe= s not require one.=0A>13/11/15 14:11:24 INFO utils.ConfigurationUtils: No e= dge output format specified. Ensure your OutputFormat does not require one.= =0A>13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is: hdf= s://localhost/user/rvesse/shortestpaths/3=0A>13/11/15 14:11:24 INFO yarn.Gi= raphYarnClient: Running Client=0A>13/11/15 14:11:24 INFO client.RMProxy: Co= nnecting to ResourceManager at /0.0.0.0:8032=0A>13/11/15 14:11:24 INFO yarn= .GiraphYarnClient: Got node report from ASM for, nodeId=3Dmbp-rvesse.home:4= 9981, nodeAddressmbp-rvesse.home:8042, nodeRackName/default-rack, nodeNumCo= ntainers0=0A>13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Defaulting per-t= ask heap size to 1024MB.=0A>13/11/15 14:11:24 INFO yarn.GiraphYarnClient: O= btained new Application ID: application_1384523910015_0004=0A>13/11/15 14:1= 1:24 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, = use mapreduce.job.id=0A>13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set t= he environment for the application master=0A>13/11/15 14:11:24 INFO yarn.Gi= raphYarnClient: Environment for AM :{CLASSPATH=3D${CLASSPATH}:./*:$HADOOP_C= ONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share= /hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOM= E/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YAR= N_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*= ,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}=0A>13/11/15 14:11:24 INF= O yarn.GiraphYarnClient: buildLocalResourceMap ....=0A>13/11/15 14:11:24 IN= FO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use= mapreduce.output.fileoutputformat.outputdir=0A>13/11/15 14:11:25 INFO yarn= .YarnUtils: Registered file in LocalResources :: hdfs://localhost/user/rves= se/giraph_yarn_jar_cache/application_1384523910015_0004/giraph-conf.xml=0A>= 13/11/15 14:11:25 INFO yarn.GiraphYarnClient: LIB JARS :/Users/rvesse/Docum= ents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT= -for-hadoop-2.2.0-jar-with-dependencies.jar=0A>13/11/15 14:11:25 INFO yarn.= YarnUtils: Class path name .=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: base= path checking .=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name = null=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null=0A>1= 3/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documen= ts/Apps/hadoop-2.2.0/etc/hadoop=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: b= ase path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop=0A>1= 3/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documen= ts/Apps/hadoop-2.2.0/share/hadoop/common/lib/*=0A>13/11/15 14:11:25 INFO ya= rn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/= share/hadoop/common/lib=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: Class pat= h name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/*=0A>1= 3/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Docu= ments/Apps/hadoop-2.2.0/share/hadoop/common=0A>13/11/15 14:11:25 INFO yarn.= YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/= hadoop/hdfs=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /U= sers/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs=0A>13/11/15 14:11= :25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoo= p-2.2.0/share/hadoop/hdfs/lib/*=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: b= ase path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hd= fs/lib=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rve= sse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/*=0A>13/11/15 14:11:25 IN= FO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2= .2.0/share/hadoop/hdfs=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: Class path= name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib/*=0A>= 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Doc= uments/Apps/hadoop-2.2.0/share/hadoop/yarn/lib=0A>13/11/15 14:11:25 INFO ya= rn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/sha= re/hadoop/yarn/*=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: base path checki= ng /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn=0A>13/11/15 = 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/= hadoop-2.2.0/share/hadoop/mapreduce/lib/*=0A>13/11/15 14:11:25 INFO yarn.Ya= rnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share= /hadoop/mapreduce/lib=0A>13/11/15 14:11:25 INFO yarn.YarnUtils: Class path = name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*=0A>= 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Doc= uments/Apps/hadoop-2.2.0/share/hadoop/mapreduce=0A>13/11/15 14:11:25 INFO y= arn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar=0A>13/11/1= 5 14:11:25 INFO yarn.YarnUtils: base path checking /contrib/capacity-schedu= ler=0A>13/11/15 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionC= ontext for GiraphApplicationMaster launch container is populated.=0A>13/11/= 15 14:11:25 INFO yarn.GiraphYarnClient: Submitting application to ASM=0A>13= /11/15 14:11:25 INFO impl.YarnClientImpl: Submitted application application= _1384523910015_0004 to ResourceManager at /0.0.0.0:8032=0A>13/11/15 14:11:2= 5 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1= 384523910015_0004=0A>13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphAp= plicationMaster container request was submitted to ResourceManager for job:= Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation=0A>13/11= /15 14:11:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples= .SimpleShortestPathsComputation, Elapsed: 0.82 secs=0A>13/11/15 14:11:26 IN= FO yarn.GiraphYarnClient: appattempt_1384523910015_0004_000001, State: ACCE= PTED, Containers used: 1=0A>13/11/15 14:11:29 ERROR yarn.GiraphYarnClient: = Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation reports F= AILED state, diagnostics show: Application application_1384523910015_0004 f= ailed 2 times due to AM Container for appattempt_1384523910015_0004_000002 = exited with =C2=A0exitCode: 1 due to: Exception from container-launch:=C2= =A0=0A>org.apache.hadoop.util.Shell$ExitCodeException:=C2=A0=0A>at org.apac= he.hadoop.util.Shell.runCommand(Shell.java:464)=0A>at org.apache.hadoop.uti= l.Shell.run(Shell.java:379)=0A>at org.apache.hadoop.util.Shell$ShellCommand= Executor.execute(Shell.java:589)=0A>at org.apache.hadoop.yarn.server.nodema= nager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.jav= a:195)=0A>at org.apache.hadoop.yarn.server.nodemanager.containermanager.lau= ncher.ContainerLaunch.call(ContainerLaunch.java:283)=0A>at org.apache.hadoo= p.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(Co= ntainerLaunch.java:79)=0A>at java.util.concurrent.FutureTask$Sync.innerRun(= FutureTask.java:303)=0A>at java.util.concurrent.FutureTask.run(FutureTask.j= ava:138)=0A>at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa= dPoolExecutor.java:895)=0A>at java.util.concurrent.ThreadPoolExecutor$Worke= r.run(ThreadPoolExecutor.java:918)=0A>at java.lang.Thread.run(Thread.java:6= 95)=0A>=0A>=0A>=0A>=0A>.Failing this attempt.. Failing the application.=0A>= 13/11/15 14:11:29 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed = cache directory for Giraph job.=0A>13/11/15 14:11:29 INFO yarn.GiraphYarnCl= ient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsCompu= tation: FAILED, total running time: 0 minutes, 3 seconds.=0A>=0A>=0A>And I = see the same error in the logs:=0A>=0A>=0A>Exception in thread "main" java.= lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMaster= =0ACaused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.Gira= phApplicationMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:20= 2) at java.security.AccessController.doPrivileged(Native Method) at java.ne= t.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoade= r.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadC= lass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java= :247)=0A>=0A>=0A>Any more ideas?=0A>=0A>=0A>Rob=0A>=0A>=0A>From: Mohammad = Islam =0A>Reply-To: =0A>Date: = Thursday, 14 November 2013 20:36=0A>To: "user@giraph.apache.org" =0A>Subject: Re: Giraph and Hadoop 2.2.0=0A>=0A>=0A>=0A>I = tried the following command few months back. I found you are not passing so= me params.=0A>>=0A>>=0A>>bin/hadoop --config etc/hadoop jar /export/home/cl= ei/giraph/giraph-1.0.0/giraph-examples/target/giraph-examples-1.?.0-for-had= oop-???-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache= .giraph.examples.SimplePageRankVertexOpt -c org.apache.giraph.combiner.Doub= leSumCombiner -w 35 -yh 20480 -yj giraph-examples-1.?.?-for-hadoop-2.??-jar= -with-dependencies.jar -vif org.apache.giraph.io.formats.JsonLongDoubleFloa= tDoubleVertexInputFormat -of org.apache.giraph.io.formats.IdWithValueTextOu= tputFormat -vip /user/me/input -op /user/me/output_35w_20G_80m_1=0A>>=0A>>= =0A>>=0A>>=0A>>On Thursday, November 14, 2013 3:22 AM, Rob Vesse wrote:=0A>> =0A>>It's the same command line I showed earlier, = just trying to run the example application:=0A>>=0A>>=0A>>mbp-rvesse:hadoop= -2.2.0 rvesse$ bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/gira= ph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with= -dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples= .SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongD= oubleFloatDoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org= .apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shor= testpaths/2 -w 1=0A>>=0A>>=0A>>Rob=0A>>=0A>>=0A>>From: Mohammad Islam =0A>>Reply-To: =0A>>Date: Thursda= y, 14 November 2013 05:57=0A>>To: "user@giraph.apache.org" =0A>>Subject: Re: Giraph and Hadoop 2.2.0=0A>>=0A>>=0A>>=0A>>What= are you running? PR?=0A>>>What is the command line?=0A>>>=0A>>>=0A>>>=0A>>= >=0A>>>=0A>>>On Wednesday, November 13, 2013 8:30 AM, Rob Vesse wrote:=0A>>> =0A>>>So I fixed the previous issue I was having w= hich I reported and submitted a patch as GIRAPH-795=0A>>>=0A>>>=0A>>>I then= went ahead with installing and configuring my external Zookeeper appropria= tely and tried running again but now the YARN application fails and I find = the following error in the logs:=0A>>>=0A>>>=0A>>>Exception in thread "main= " java.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationM= aster=0ACaused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn= .GiraphApplicationMaster at java.net.URLClassLoader$1.run(URLClassLoader.ja= va:202) at java.security.AccessController.doPrivileged(Native Method) at ja= va.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.Class= Loader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.= loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader= .java:247)=0A>>>=0A>>>=0A>>>Anybody have idea what the cause of this one mi= ght be or how to fix it?=0A>>>=0A>>>=0A>>>Cheers,=0A>>>=0A>>>=0A>>>Rob=0A>>= >=0A>>>=0A>>>From: Rob Vesse =0A>>>Reply-To: =0A>>>Date: Wednesday, 13 November 2013 15:35=0A>>>To: = =0A>>>Subject: Re: Giraph and Hadoop 2.2.0=0A>>>= =0A>>>=0A>>>=0A>>>Mohammad=0A>>>>=0A>>>>=0A>>>>This got me a build that at = least attempts to start to run but not I see the following:=0A>>>>=0A>>>>= =0A>>>>mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop jar /Users/rvesse/Documen= ts/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-f= or-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner or= g.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.gir= aph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/rvesse= /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputForm= at -op /user/rvesse/shortestpaths/2 -w 1=0A>>>>2013-11-13 15:13:29.908 java= [17402:1203] Unable to load realm info from SCDynamicStore=0A>>>>13/11/13 1= 5:13:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library fo= r your platform... using builtin-java classes where applicable=0A>>>>13/11/= 13 15:13:31 INFO utils.ConfigurationUtils: No edge input format specified. = Ensure your InputFormat does not require one.=0A>>>>13/11/13 15:13:31 INFO = utils.ConfigurationUtils: No edge output format specified. Ensure your Outp= utFormat does not require one.=0A>>>>13/11/13 15:13:31 INFO yarn.GiraphYarn= Client: Final output path is: hdfs://localhost/user/rvesse/shortestpaths/2= =0A>>>>Exception in thread "main" java.lang.NullPointerException=0A>>>>at o= rg.apache.giraph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(Gira= phYarnClient.java:460)=0A>>>>at org.apache.giraph.yarn.GiraphYarnClient.run= (GiraphYarnClient.java:116)=0A>>>>at org.apache.giraph.GiraphRunner.run(Gir= aphRunner.java:96)=0A>>>>at org.apache.hadoop.util.ToolRunner.run(ToolRunne= r.java:70)=0A>>>>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:8= 4)=0A>>>>at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)=0A>>= >>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)=0A>>>>at s= un.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39= )=0A>>>>at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25)=0A>>>>at java.lang.reflect.Method.invoke(Method.java:= 597)=0A>>>>at org.apache.hadoop.util.RunJar.main(RunJar.java:212)=0A>>>>=0A= >>>>=0A>>>>Which indirectly implies that I have some missing configuration = to run Giraph on YARN. =C2=A0I don't think that should be a NPE and it look= s to be a trivial fix so I'll try and fix and send a pull request for that.= =0A>>>>=0A>>>>=0A>>>>The error message I should receive if the NPE wasn't h= appening is the following:=0A>>>>=0A>>>>=0A>>>>Giraph on YARN does not curr= entlysupport Giraph-managed ZK instances: use a standalone ZooKeeper.=0A>>>= >=0A>>>>=0A>>>>What configuration options do I need to have set to point Gi= raph at an external Zookeeper?=0A>>>>=0A>>>>=0A>>>>Cheers,=0A>>>>=0A>>>>=0A= >>>>Rob=0A>>>>=0A>>>>=0A>>>>From: Mohammad Islam =0A>>= >>Reply-To: =0A>>>>Date: Wednesday, 13 November 2= 013 11:15=0A>>>>To: "user@giraph.apache.org" =0A>>= >>Subject: Re: Giraph and Hadoop 2.2.0=0A>>>>=0A>>>>=0A>>>>=0A>>>>Hi Rob,= =0A>>>>>Did you try with this command "mvn -Phadoop_yarn -Dhadoop.version= =3D2.2.0 clean package -DskipTests"?=0A>>>>>Regards,=0A>>>>>Mohammad=0A>>>>= >=0A>>>>>=0A>>>>>=0A>>>>>On Wednesday, November 13, 2013 2:55 AM, Rob Vesse= wrote:=0A>>>>> =0A>>>>>Hi all=0A>>>>>=0A>>>>>=0A>>>= >>I'm having trouble getting Giraph to build and run against a Hadoop 2.2.0= installation. =C2=A0I tried the hadoop_2.0.3 profile with no luck because = I get class cast errors when trying to run the examples.=0A>>>>>=0A>>>>>=0A= >>>>>Is there a specific profile I should be building to support Hadoop 2.2= .0 or is this not supported yet?=0A>>>>>=0A>>>>>=0A>>>>>Thanks,=0A>>>>>=0A>= >>>>=0A>>>>>Rob=0A>>>>>=0A>>>>>=0A>>>=0A>>>=0A>>=0A>> ---887950874-152249639-1384813456=:4648 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
I used the= following shell script to run PR where my current working directory contai= ns the graph*.jar in Hadoop 2.1.1.
. You can= try something similar.



HAD_VERSION=3D2.1.1-SNAPSHOT<= /div>
hadoop fs -rmr out
export GIRAPH_HOME=3D/home/= mislam/softwares/giraph
#export GIRAPH_CORE=3D$GIRAPH_HOME/giraph-core/target/giraph-1.1.0-SNAPSHO= T-for-hadoop-$HAD_VERSION-jar-with-dependencies.jar
#export GIRAPH_EXAMPLE=3D$GIRAPH_HOME/giraph-e= xamples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_VERSION-jar-w= ith-dependencies.jar

CLASSPATH=3D$CLASSPATH:$GIRAPH_H= OME/conf
#CLASSPATH=3D$= CLASSPATH:$GIRAPH_CORE
= #CLASSPATH=3D$CLASSPATH:$GIRAPH_EXAMPLE

e= xport CLASSPATH=3D$CLASSPATH
export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$CLASSPATH

echo $HADOOP_CLASSPATH

hadoop --config = $HADOOP_HOME/etc/hadoop jar giraph-examples-1.1.0-SNAPSHOT-for-hadoop-$HAD_= VERSION-jar-with-dependencies.jar  org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimplePageRankComputation -c org.apache.giraph.= combiner.DoubleSumCombiner -w 1 -yh 1024 -yj giraph-examples-1.1.0-SNAPSHOT= -for-hadoop-$HAD_VERSION-jar-with-dependencies.jar -vif org.apache.giraph.i= o.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vof org.apache.giraph= .io.formats.IdWithValueTextOutputFormat -mc org.apache.giraph.examples.Simp= lePageRankComputation\$SimplePageRankMasterCompute -vip input -op out
=

On Monday, November 18, 2013= 2:24 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
=
Hey all=

Bumping this threa= d because I am still completely blocked on this and hoping someone can help= me.  As explained earlier in the thread I have Giraph built for pure = YARN on stable Hadoop 2.2.0 release like so:

mvn =E2=80=93Phadoop_yarn =E2=80=93Dhadoop.version=3D2.2.0 = =E2=80=93DskipTests=3Dtrue clean install

But when I try and run the simple shortest paths demo like so:

$= > bin/hadoop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-example= s/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependenc= ies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleSho= rtestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloat= DoubleVertexInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.gi= raph.io.formats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths = -w 1 -yj /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/gi= raph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar

The application fails and the YARN logs show = the following error:

Exception in thread "main" j=
ava.lang.NoClassDefFoundError: org/apache/giraph/yarn/GiraphApplicationMast=
er=0ACaused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.Gi=
raphApplicationMaster=0A=09at java.net.URLClassLoader$1.run(URLClassLoader.=
java:202)=0A=09at java.security.AccessController.doPrivileged(Native Method=
)=0A=09at java.net.URLClassLoader.findClass(URLClassLoader.java:190)=0A=09a=
t java.lang.ClassLoader.loadClass(ClassLoader.java:306)=0A=09at sun.misc.La=
uncher$AppClassLoader.loadClass(Launcher.java:301)=0A=09at java.lang.ClassL=
oader.loadClass(ClassLoader.java:247)

Any ideas why the GiraphApplicationMaster would not be fou=
nd?  Have I just got something basic wrong in my configurati=
on or is this a bug in Giraph?

I found an earlier thread on the same topic (http=
://mail-archives.apache.org/mod_mbox/giraph-user/201307.mbox/%3c51E5A4D1.40=
006@googlemail.com%3e) but it was never resolved hence why I =
have now changed the topic in the hope it will jog someone's memory<=
/pre>

Cheers,

Ro=
b


Adding the =E2=80=93yj argume= nt to specify the YARN JARs doesn't make any difference.

Here is the complete console output fr= om my run:

bin/hadoop jar /Users/rve= sse/Documents/Work/Code/giraph/giraph-examples/target/giraph-examples-1.1.0= -SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.Gira= phRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org= .apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /= user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTex= tOutputFormat -op /user/rvesse/shortestpaths/3 -w 1 -yj /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-exa= mples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
2013-11-15 14:11:22.897 java[4877:1203] Unabl= e to load realm info from SCDynamicStore
13/11/15 14:11:22 WARN util.NativeCodeLoader: Unable to load native-= hadoop library for your platform... using builtin-java classes where applic= able
13/11/15 14:11:24 INFO utils.= ConfigurationUtils: No edge input format specified. Ensure your InputFormat= does not require one.
13/11/15 14= :11:24 INFO utils.ConfigurationUtils: No edge output format specified. Ensu= re your OutputFormat does not require one.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Final output path is= : hdfs://localhost/user/rvesse/shortestpaths/3
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Running Cl= ient
13/11/15 14:11:24 INFO client= .RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
<= font face=3D"Consolas">13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Got no= de report from ASM for, nodeId=3Dmbp-rvesse.home:49981, nodeAddressmbp-rves= se.home:8042, nodeRackName/default-rack, nodeNumContainers0
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Def= aulting per-task heap size to 1024MB.
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Obtained new Application = ID: application_1384523910015_0004
13/11/15 14:11:24 INFO Configuration.deprecation: mapred.job.id is depreca= ted. Instead, use mapreduce.job.id
13/11/15 14:11:24 INFO yarn.GiraphYarnClient: Set the environment for the application master
13/11/15 14:11= :24 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=3D${CLASSPAT= H}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_C= OMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:= $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/ya= rn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/ha= doop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
13/11/15 14:11:24 INFO yarn.GiraphYarnClie= nt: buildLocalResourceMap ....
13/= 11/15 14:11:24 INFO Configuration.deprecation: mapred.output.dir is depreca= ted. Instead, use mapreduce.output.fileoutputformat.outputdir
<= div>13/11/15 14:11:25 INFO yarn.YarnUtils: Register= ed file in LocalResources :: hdfs://localhost/user/rvesse/giraph_yarn_jar_cache/application_13845239100= 15_0004/giraph-conf.xml
13/11/15 1= 4:11:25 INFO yarn.GiraphYarnClient: LIB JARS :/Users/rvesse/Documents/Work/= Code/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoo= p-2.2.0-jar-with-dependencies.jar
= 13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name .
<= font face=3D"Consolas">13/11/15 14:11:25 INFO yarn.YarnUtils: base path che= cking .
13/11/15 14:11:25 INFO yar= n.YarnUtils: Class path name null
= 13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking null
=
13/11/15 14:11:25 INFO yarn.YarnUtils: Class p= ath name /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
<= div>13/11/15 14:11:25 INFO yarn.YarnUtils: base pat= h checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/etc/hadoop
=
13/11/15 14:11:25 INFO yarn.YarnUtils: Class p= ath name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/common/lib/= *
13/11/15 14:11:25 INFO yarn.Yarn= Utils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/h= adoop/common/lib
13/11/15 14:11:25= INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2= .2.0/share/hadoop/common/*
13/11/1= 5 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/= Apps/hadoop-2.2.0/share/hadoop/common
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Do= cuments/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs
13/11/15 14:11:25 INFO yarn.YarnUtils: Class pat= h name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/hdfs/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtil= s: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoo= p/hdfs/lib
13/11/15 14:11:25 INFO = yarn.YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/s= hare/hadoop/hdfs/*
13/11/15 14:11:= 25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/had= oop-2.2.0/share/hadoop/hdfs
13/11/= 15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvesse/Documents/Ap= ps/hadoop-2.2.0/share/hadoop/yarn/lib/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class= path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/yarn/*
13/11/15 14:11:25 INFO yarn.YarnUtil= s: base path checking /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoo= p/yarn
13/11/15 14:11:25 INFO yarn= .YarnUtils: Class path name /Users/rvesse/Documents/Apps/hadoop-2.2.0/share= /hadoop/mapreduce/lib/*
13/11/15 1= 4:11:25 INFO yarn.YarnUtils: base path checking /Users/rvesse/Documents/App= s/hadoop-2.2.0/share/hadoop/mapreduce/lib
13/11/15 14:11:25 INFO yarn.YarnUtils: Class path name /Users/rvess= e/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce/*
13/11/15 14:11:25 INFO yarn.YarnUtils: base path check= ing /Users/rvesse/Documents/Apps/hadoop-2.2.0/share/hadoop/mapreduce
13/11/15 14:11:25 INFO yarn.YarnUtils: Clas= s path name /contrib/capacity-scheduler/*.jar
13/11/15 14:11:25 INFO yarn.YarnUtils: base path checking /co= ntrib/capacity-scheduler
13/11/15 = 14:11:25 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for Girap= hApplicationMaster launch container is populated.
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Submitting ap= plication to ASM
13/11/15 14:11:25= INFO impl.YarnClientImpl: Submitted application application_1384523910015_= 0004 to ResourceManager at /0.0.0.0:8032
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: Got new appId after su= bmission :application_1384523910015_0004
13/11/15 14:11:25 INFO yarn.GiraphYarnClient: GiraphAppl= icationMaster container request was submitted to ResourceManager for job: G= iraph: org.apache.giraph.examples.SimpleShortestPathsComputation
13/11/15 14:11:26 INFO yarn.GiraphYarnClient= : Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation, Elapse= d: 0.82 secs
13/11/15 14:11:26 INF= O yarn.GiraphYarnClient: appattempt_1384523910015_0004_000001, State: ACCEP= TED, Containers used: 1
13/11/15 1= 4:11:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.Sim= pleShortestPathsComputation reports FAILED state, diagnostics show: Applica= tion application_1384523910015_0004 failed 2 times due to AM Container for = appattempt_1384523910015_0004_000002 exited with  exitCode: 1 due to: = Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: 
=09at org.apache.hadoop.util.Shel= l.runCommand(Shell.java:464)
=09at org.apache.hadoop.util.Shell.run(Shell.java:379)
=09at org.apache.hadoop.util.Shell$ShellCommandExecut= or.execute(Shell.java:589)
=09a= t org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch= Container(DefaultContainerExecutor.java:195)
=09at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Contai= nerLaunch.call(ContainerLaunch.java:283)
=09at org.apache.hadoop.yarn.server.nodemanager.containermanager.= launcher.ContainerLaunch.call(ContainerLaunch.java:79)
=09at java.util.concurrent.FutureTask$Sync.innerRun= (FutureTask.java:303)
=09at ja= va.util.concurrent.FutureTask.run(FutureTask.java:138)
=09at java.util.concurrent.ThreadPoolExecutor$Worke= r.runTask(ThreadPoolExecutor.java:895)
=09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:918)
=09at java.lang.Thread= .run(Thread.java:695)


=
.Failing this attempt.. Failing the appl= ication.
13/11/15 14:11:29 INFO ya= rn.GiraphYarnClient: Cleaning up HDFS distributed cache directory for Girap= h job.
13/11/15 14:11:29 INFO yarn= .GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShort= estPathsComputation: FAILED, total running time: 0 minutes, 3 seconds.

And I see the same error in t= he logs:

Exception in thread "main" java.lang.NoClassDefFoundError: or=
g/apache/giraph/yarn/GiraphApplicationMaster=0ACaused by: java.lang.ClassNo=
tFoundException: org.apache.giraph.yarn.GiraphApplicationMaster=0A=09at jav=
a.net.URLClassLoader$1.run(URLClassLoader.java:202)=0A=09at java.security.A=
ccessController.doPrivileged(Native Method)=0A=09at java.net.URLClassLoader=
.findClass(URLClassLoader.java:190)=0A=09at java.lang.ClassLoader.loadClass=
(ClassLoader.java:306)=0A=09at sun.misc.Launcher$AppClassLoader.loadClass(L=
auncher.java:301)=0A=09at java.lang.ClassLoader.loadClass(ClassLoader.java:=
247)

Any more idea=
s?

Rob


I tried the foll= owing command few months back. I found you are not passing some params.

=

On Thursday, November 14, 201= 3 3:22 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
It's the same command line I showed earlier, just trying to=0A run = the example application:

mbp-rvesse:hadoop-2.2.0 rvesse$ bin/ha= doop jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/gi= raph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org= .apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsC= omputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVerte= xInputFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.for= mats.IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/2 -w 1

=
Rob

From: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
<= span style=3D"font-weight:bold;">Date: Thursday, 14 November 2013 0= 5:57
To: "user@giraph.apache.org" <user@giraph.apache.org>Subject: Re: Gira= ph and Hadoop 2.2.0

<= blockquote id=3D"yiv5676927396MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style=3D"= BORDER-LEFT:#b5c4df 5 solid;PADDING:0 0 0 5;MARGIN:0 0 0 5;">
What are you running? PR?<= /div>
What is the command line?

=

On Wednesday, November 13, 2013 = 8:30 AM, Rob Vesse <rvesse@dotnetrdf.org> wrote:
So I fixed the previous issue I was having which I reported and su= bmitted a patch as GIRAPH-795

I then went ahead with installing and configuring my external Zoo= keeper appropriately and tried running again but now the YARN application f= ails and I find the following error in the logs:

Exception in thread "mai=
n" java.lang.NoClassDefFoundError:=0A org/apache/giraph/yarn/GiraphApplicat=
ionMaster=0ACaused by: java.lang.ClassNotFoundException: org.apache.giraph.=
yarn.GiraphApplicationMaster=0A=09at java.net.URLClassLoader$1.run(URLClass=
Loader.java:202)=0A=09at java.security.AccessController.doPrivileged(Native=
 Method)=0A=09at java.net.URLClassLoader.findClass(URLClassLoader.java:190)=
=0A=09at java.lang.ClassLoader.loadClass(ClassLoader.java:306)=0A=09at sun.=
misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)=0A=09at java.lang=
.ClassLoader.loadClass(ClassLoader.java:247)

Anybody have idea what the cause of this one mig=
ht be or how to fix it?

Cheers,

Rob

From: Rob Vesse <rvesse@dotnetrdf.org>
Reply-To: <user@giraph.apache.org>
Date: Wednesday, 13= November 2013 15:35
To= : <us= er@giraph.apache.org>
Subject: Re: Giraph and Hadoop 2.2.0
=

Mohammad

This got me a build that at least atte= mpts to start to run but not I see the following:

mbp-rvesse:hadoop-2.2.0 rvesse$ bin/hadoop = jar /Users/rvesse/Documents/Work/Code/giraph/giraph-examples/target/giraph-= examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apac= he.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComput= ation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInpu= tFormat -vip /user/rvesse/tiny_graph.txt -vof org.apache.giraph.io.formats.= IdWithValueTextOutputFormat -op /user/rvesse/shortestpaths/2 -w 1
2013-11-13 15:13:29.908 java[17402:1203] Unable to load realm info from SC= DynamicStore
13/11/13 15:13:29 WARN util.NativeCodeLoader: Unable= to load native-hadoop library for your platform... using builtin-java clas= ses where applicable
13/11/13 15:13:31 INFO utils.ConfigurationUt= ils: No edge input format specified. Ensure your InputFormat does not requi= re one.
13/11/13 15:13:31 INFO utils.ConfigurationUtils: No=0A ed= ge output format specified. Ensure your OutputFormat does not require one.<= /div>
13/11/13 15:13:31 INFO yarn.GiraphYarnClient: Final output path i= s: hdfs://localhost/user/rvesse/shortestpaths/2
Exception in thre= ad "main" java.lang.NullPointerException
=09at org.apache.gira= ph.yarn.GiraphYarnClient.checkJobLocalZooKeeperSupported(GiraphYarnClient.j= ava:460)
=09at org.apache.giraph.yarn.GiraphYarnClient.run(Gir= aphYarnClient.java:116)
=09at org.apache.giraph.GiraphRunner.r= un(GiraphRunner.java:96)
=09at org.apache.hadoop.util.ToolRunn= er.run(ToolRunner.java:70)
=09at org.apache.hadoop.util.ToolRunner.= run(ToolRunner.java:84)
=09at org.apache.giraph.GiraphRunner.m= ain(GiraphRunner.java:126)
=09at sun.reflect.NativeMethodAcces= sorImpl.invoke0(Native Method)
=09at sun.reflect.NativeMethodA= ccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
=09at su= n.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.= java:25)
=09at java.lang.reflect.Method.invoke(Method.java:597= )
=09at=0A org.apache.hadoop.util.RunJar.main(RunJar.java:212)=

Which indirectly implies tha= t I have some missing configuration to run Giraph on YARN.  I don't th= ink that should be a NPE and it looks to be a trivial fix so I'll try and f= ix and send a pull request for that.

The error message I should receive if the NPE wasn't happening is the fol= lowing:

Giraph on YARN does not cur= rentlysupport Giraph-managed ZK instances: use a standalone ZooKeeper.

What configuration options do I need to= have set to point Giraph at an external Zookeeper?

Cheers,

Rob
<= div>
Fro= m: Mohammad Islam <mislam77@yahoo.com>
Reply-To: <user@giraph.apache.org>
Date: Wednesday, 13 November 2013 11:15To: "user@giraph.apache.org" = <user@giraph.apache.org>
Subject: Re: Giraph= and Hadoop 2.2.0

Hi Rob,
Did you try with this command "mvn -= Phadoop_yarn -Dhadoop.version=3D2.2.0 clean package -DskipTests"?
Regards,
Mohammad


= On Wednesday, November 13, 2013 2:55 AM, R= ob Vesse <rvesse@= dotnetrdf.org>=0A wrote:
Hi = all

I'm having trouble getting Gira= ph to build and run against a Hadoop 2.2.0 installation.  I tried the = hadoop_2.0.3 profile with no luck because I get class cast errors when tryi= ng to run the examples.

Is there a = specific profile I should be building to support Hadoop 2.2.0 or is this no= t supported=0A yet?

Thanks,

Rob





= =

---887950874-152249639-1384813456=:4648--