Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C31A110DED for ; Thu, 20 Nov 2014 19:33:46 +0000 (UTC) Received: (qmail 19112 invoked by uid 500); 20 Nov 2014 19:33:46 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 19059 invoked by uid 500); 20 Nov 2014 19:33:46 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 19049 invoked by uid 99); 20 Nov 2014 19:33:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Nov 2014 19:33:46 +0000 X-ASF-Spam-Status: No, hits=3.1 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [212.82.97.76] (HELO nm29-vm7.bullet.mail.ir2.yahoo.com) (212.82.97.76) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Nov 2014 19:33:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.it; s=s2048; t=1416511936; bh=XHHO0NyyRtxGoRjl5g3SFQC0gB4pD3xwis5aonqO7ks=; h=Subject:References:From:In-Reply-To:Date:To:From:Subject; b=ljz7NbnKuylkSgkH4L+dg83VTKm7OknNnlk7rnG1+TpNxSN7GBTDkf595L1psRNQ1uaemme1W/slV6aibh93dDV3rpCvKLHuwInwM2GhJYW+HhXyS/F946KGPcpAcDRWUAPZMZv/nsK8yltpSYo/xizcSf8YbkL14+AoEA8P89PI6oxpiQ94M13OUso+j+i5AMbfR/kbOx3EqfSG/OKl+jFYBpGh/aIzUsgRMtZ7Wdtfdnh9PnCTb4T9RIvtjdmUMWasyOTKpsqLylmz1I1PvIccpK3RkHJZXsSlG3jjHRg5m/kH++VpfAesgu3hFq+Qe4OZMZDmpPEF1TEeietvHw== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.it; b=nP19qWheSDEEKSrdv9PszTsH6pTZ34bl5QIxLy9cKRYKUMc8zMLRo7W1OeHanSX1UJSLPRMN9icoJZ0iYQ1dvQPUd6ix4AQRyGcFKVxyq3Y5eL5bkCF7tpGhQqnLLDBW5f76axU0QOleshRONACkaDtKNLWko1ctUxwfa2U3MgMkyaRNoYtlAP9SDKbajs+/vxDIIMAyI+ngKBxm+QHjgJWdoiaLkVtKGB82ZVpxH6a2RxFyyQGDXrxCoqoC8H2odRy68oygXL0mM16f4/01jKVl1MtXPxXvSsNhba8KU/1BalUEWPJ2RyNUTEqD45pqglKDEImq6vI+NXosMvVmIA==; Received: from [212.82.98.58] by nm29.bullet.mail.ir2.yahoo.com with NNFMP; 20 Nov 2014 19:32:16 -0000 Received: from [46.228.39.66] by tm11.bullet.mail.ir2.yahoo.com with NNFMP; 20 Nov 2014 19:32:16 -0000 Received: from [127.0.0.1] by smtp103.mail.ir2.yahoo.com with NNFMP; 20 Nov 2014 19:32:16 -0000 X-Yahoo-Newman-Id: 338324.42464.bm@smtp103.mail.ir2.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: VBD5CvIVM1kWW0aUEHbb1M9nfcHyodjkeDF8bJUNvtYpYPN zeh93IbiYrcOLwEOJcYAgWt5MPWnJM22SED6rbrXnh5u7pTPpgFbiCBPd9o4 Do1_sK1VmPE1FZdnYte.yRLNELcheeTIPLeLPEIlFZw32JordD1mH6arDyE7 BuxvF3R3u9Y8l_b7h7EfnHzJmlbTxzRMLWOVUsm5afLAD__kLDy9QQP_nmJW uCWK0lGrxvyGUyK706zUVBqYJG_gvHSlA4aESCIEKdPbKaGY3BydjJFKQ0Kw oQOGY.CnABSwWAmDiHigPl18edQi7z_Efl6gp_MVqSpQO4nWij32np1JANSw uLKmXHlDzPxFXkJLHbZGyCO65QHsQPLDaiPysvMuyLmnS6qMg2Ur_DVgy0Sa xaQdZQ4ELwzyy.Eitqg1Xc6Kf.kRoEVr3vDLtmBYcQ0H84uW5sSBqyZ_lshC OkwUFtIviNv3_iANNDGzKRpmZR5masSgkAPCgyZJgphNr4uBMXZhb4pKTQzL RE39egrp0.Uu8JB.UTKMU89RAc50qv2AXHnNcWyGwFCnL8cmwczXltZwDeB9 0D2w95fr_Bqd1P_WB0U35xRdQCbINDC7gO_hU5Pb_oUnipSo_RazYPUUpJ.c TtGYsn9TlzNY320lFHPbhWqOKBNNJeiCvQ9GpUuvjANd8E8541Ior6GdJJJT aL1Va6XZh.MrBBOnqGmnqIl5nTBm2VBEVXrayTg0gDbpwOHODDt6MM5mI7lk Imt3JAOcDlPSikFlzMY.ZjuC_BR9iqKXtp17g5xKCqS3hJEdYbAIcbZf2T8M NYP_POy.F3lxKmIs3z0XkDWZIyw-- X-Yahoo-SMTP: qoe0kIaswBCBQnIixtWAKnaaI7JzLFw- Subject: Re: Help with Giraph on Yarn References: <77A73EB3-7967-4AB4-B5E6-191B46898119@yahoo.it> <0B32BE05-B6B6-4FEF-B2E8-8B99A45F4A83@yahoo.it> <41384542-C90C-4573-992E-CDD5E29B5803@yahoo-inc.com> <536AC63F-4C87-4071-A1D9-D867DCC6769B@yahoo.it> <92F1CFC1-0961-49C3-AE1F-FE970BFA4405@yahoo.it> From: Alessandro Negro Content-Type: multipart/alternative; boundary=Apple-Mail-BC7C39D7-1E69-422E-AE56-5745D70FA6F7 X-Mailer: iPhone Mail (11D257) In-Reply-To: Message-Id: <89D518FA-00B2-40C4-B391-BDCFB6CA2B50@yahoo.it> Date: Thu, 20 Nov 2014 20:32:12 +0100 To: "user@giraph.apache.org" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-BC7C39D7-1E69-422E-AE56-5745D70FA6F7 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Hi Das, I mean as user you run Hadoop yarn resource manager. Alessandro > Il giorno 20/nov/2014, alle ore 19:22, D Adams ha scr= itto: >=20 > Alessandro, > I'm not sure what you mean, should I create a new user on my system n= amed 'yarn'? I'm new to both hadoop and giraph, so I'm not sure. >=20 > Tripti, > I'll get those logs as soon as I can. >=20 > Thank you both, >=20 > V/r > Das >=20 >> On Nov 20, 2014 3:13 AM, "Alessandro Negro" wrote: >> Hi Tripti, >> I agree that a more detailed error log could be useful. >>=20 >> Thanks, >> Alessandro >>=20 >>=20 >>=20 >>=20 >> Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh ha scritto: >>=20 >> > --_000_D092AB9C1CB42triptiyahooinccom_ >> > Content-Type: text/plain; charset=3D"Windows-1252" >> > Content-Transfer-Encoding: quoted-printable >> > >> > Hi Alessandro, >> > >> > I guess u r talking about the documentation on Giraph webpage which men= tion=3D >> > s addition of this new option which is more or less mandatory for yarn b= ase=3D >> > d profiles. >> > When I first ran the Giraph on yarn without the proper =3D96yj option, t= here =3D >> > was no way I could figure out from the Application/Container logs that t= his=3D >> > was the issue. >> > I think, it=3D92ll be a good idea to have this message in the logs for e= asy d=3D >> > ebugging. >> > >> > Thanks, >> > Tripti. >> > >> > >> > From: Alessandro Negro = > >> > Reply-To: "user@giraph.apache.org" > > aph.apache.org> >> > Date: Tuesday, November 18, 2014 at 4:31 PM >> > To: "user@giraph.apache.org" > > ache.org> >> > Subject: Re: Help with Giraph on Yarn >> > >> > Hi Eli, >> > no I think that the help message is clear enough, generally when I read= =3D93=3D >> > jar=3D94 I always mean absolute path for jar, but this is a my personal= misun=3D >> > derstanding. >> > >> > Thanks, >> > Alessandro >> > >> > >> > Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman > > m> ha scritto: >> > >> > Sounds like you got it figured out since last time I checked this list,= sor=3D >> > ry it was a pain. Feel free to drop a JIRA ticket if we can make help m= essa=3D >> > ge etc. for -yj option more clear, there's lots to improve there. >> > >> > On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro > > o:alenegro81@yahoo.it>> wrote: >> > Hi Tripti, >> > finally I was able to run the test with success. It was an issue of per= miss=3D >> > ion since I was running as ale not as yarn. >> > Let me say that now I=3D92m able to run graph examples on Yarn 2.5.1. T= his is=3D >> > the final result: >> > >> > 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apa= che.=3D >> > giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total runnin= g ti=3D >> > me: 0 minutes, 21 seconds. >> > >> > >> > Many thanks for your support, >> > Alessandro >> > >> > Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh > > ailto:tripti@yahoo-inc.com>> ha scritto: >> > >> > I don't know if u have access to this node. But if u do, u could check i= f t=3D >> > he file is indeed there and u have access to it. >> > >> > Sent from my iPhone >> > >> > On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" > > alenegro81@yahoo.it>> wrote: >> > >> > You are right it works, but now I receive the following error: >> > >> > SLF4J: Class path contains multiple SLF4J bindings. >> > SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir= /use=3D >> > rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-= exam=3D >> > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf= 4j/i=3D >> > mpl/StaticLoggerBinder.class] >> > SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/c= ommo=3D >> > n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]= >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an exp= lana=3D >> > tion. >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> > 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from= SCD=3D >> > ynamicStore >> > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: C= oul=3D >> > d not configure the containerlaunch context for GiraphYarnTasks. >> > at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Gi= raph=3D >> > ApplicationMaster.java:391) >> > at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphAppl= icat=3D >> > ionMaster.java:78) >> > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnab= le.b=3D >> > uildContainerLaunchContext(GiraphApplicationMaster.java:522) >> > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnab= le.r=3D >> > un(GiraphApplicationMaster.java:479) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .jav=3D >> > a:1145) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.ja=3D >> > va:615) >> > at java.lang.Thread.run(Thread.java:744) >> > Caused by: java.io.FileNotFoundException: File does not exist: hdfs://h= adoo=3D >> > p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937= _000=3D >> > 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies= .jar >> > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFi= leSy=3D >> > stem.java:1072) >> > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFi= leSy=3D >> > stem.java:1064) >> > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkRe= solv=3D >> > er.java:81) >> > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Distribut= edFi=3D >> > leSystem.java:1064) >> > at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java= :153=3D >> > ) >> > at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:= 77) >> > at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Gi= raph=3D >> > ApplicationMaster.java:387) >> > ... 6 more >> > >> > >> > That justify the other error I receive in the task: >> > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask= >> > >> > Thanks, >> > >> > Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh > > ailto:tripti@yahoo-inc.com>> ha scritto: >> > >> > Why r u adding two jars? Example jar ideally contains core library so e= very=3D >> > thing should be available with just one example jar included >> > >> > Sent from my iPhone >> > >> > On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" > > alenegro81@yahoo.it>> wrote: >> > >> > Hi, >> > now it seems better, I need to add: >> > >> > giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph= -exa=3D >> > mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> > >> > >> > Now it seems that after a lot of cycle it fail with this error: >> > >> > Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask= >> > >> > But in this case the error appear in task-3-stderr.log not in gam-stde= rr.l=3D >> > og where there is the following error: >> > >> > LF4J: Class path contains multiple SLF4J bindings. >> > SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir= /use=3D >> > rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-= 1.1.=3D >> > 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/S= tati=3D >> > cLoggerBinder.class] >> > SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir= /use=3D >> > rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-= exam=3D >> > ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf= 4j/i=3D >> > mpl/StaticLoggerBinder.class] >> > SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/c= ommo=3D >> > n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]= >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an exp= lana=3D >> > tion. >> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >> > 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from S= CDy=3D >> > namicStore >> > Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: C= oul=3D >> > d not configure the containerlaunch context for GiraphYarnTasks. >> > at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Gi= raph=3D >> > ApplicationMaster.java:391) >> > at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphAppl= icat=3D >> > ionMaster.java:78) >> > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnab= le.b=3D >> > uildContainerLaunchContext(GiraphApplicationMaster.java:522) >> > at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnab= le.r=3D >> > un(GiraphApplicationMaster.java:479) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .jav=3D >> > a:1145) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.ja=3D >> > va:615) >> > at java.lang.Thread.run(Thread.java:744) >> > Caused by: java.io.FileNotFoundException: File does not exist: hdfs://h= adoo=3D >> > p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937= _000=3D >> > 6/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFi= leSy=3D >> > stem.java:1072) >> > at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFi= leSy=3D >> > stem.java:1064) >> > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkRe= solv=3D >> > er.java:81) >> > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Distribut= edFi=3D >> > leSystem.java:1064) >> > at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java= :153=3D >> > ) >> > at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:= 77) >> > at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Gi= raph=3D >> > ApplicationMaster.java:387) >> > ... 6 more >> > >> > >> > Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh > > ailto:tripti@yahoo-inc.com>> ha scritto: >> > >> > Hi, >> > I remember seeing this thread, but since i was already deep into fixing= thi=3D >> > s issue in production, I didn't really wait for the patch, not even the= mun=3D >> > ge symbol as I was not even able to compile the yarn profile( sorry for= bei=3D >> > ng so ruthless to giraph :-( ) >> > I did later cross check with the patch and it was quite similar to my c= hang=3D >> > e but sure it wouldn't work for all profiles(in fact some profiles may e= ven=3D >> > break due to Hadoop class incompatibility ) I can share those changes w= ith=3D >> > u once I get back to my computer but I don't think that could have caus= ed =3D >> > this problem. >> > Just try the small changed I mentioned in my previous reply and let me k= now=3D >> > if that helped. >> > >> > Thanks, >> > Tripti >> > Sent from my iPhone >> > >> > On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" > > alenegro81@yahoo.it>> wrote: >> > >> > Hi Tripti, >> > let me add one info, As you can see from this thread of email: >> > >> > http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3CC3F7= 5A9=3D >> > 8-BDE8-4205-971F-1F7241E6E83F@yahoo.it%3E >> > >> > I had some issue while compiling for Hadoop 2.5.1 could be that into th= e pr=3D >> > ocess I loose something? >> > >> > Thanks again, >> > Alessandro >> > >> > >> > >> > Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro > > t> ha scritto: >> > >> > Hi Tripti, >> > yes I run in this way: >> > >> > >> > $HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples= /tar=3D >> > get/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependenci= es.j=3D >> > ar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShor= test=3D >> > PathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatD= oubl=3D >> > eVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.forma= ts.I=3D >> > dWithValueTextOutputFormat -op /shortestpaths -w 1 -yj /Users/ale/extpr= j/gi=3D >> > raph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with= -dep=3D >> > endencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-ex= ampl=3D >> > es-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> > >> > And this is the complete trace: >> > >> > 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from S= CDy=3D >> > namicStore >> > 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-had= oop =3D >> > library for your platform... using builtin-java classes where applicabl= e >> > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format s= peci=3D >> > fied. Ensure your InputFormat does not require one. >> > 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format s= pec=3D >> > ified. Ensure your OutputFormat does not require one. >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdf= s://=3D >> > hadoop-master:9000/shortestpaths >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client >> > 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at= /0.=3D >> > 0.0.0:8032 >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM f= or,=3D >> > nodeId=3D3D10.61.137.28:59677, nodeAddress 1= 0.61=3D >> > .137.28:8042, nodeRackName /default-rack, no= deNu=3D >> > mContainers 0 >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap s= ize=3D >> > to 1024MB. >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application I= D: =3D >> > application_1415264041937_0005 >> > 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id> > ed.job.id/> is deprecated. Instead, use mapreduce.job.id> > ob.id/> >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for t= he a=3D >> > pplication master >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLAS= SPAT=3D >> > H=3D3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoo= p/comm=3D >> > on/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/sh= are/=3D >> > hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HO= ME/s=3D >> > hare/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MA= PRED=3D >> > _HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapredu= ce/l=3D >> > ib/*} >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap ...= . >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResource= s ::=3D >> > hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_14= 152=3D >> > 64041937_0005/giraph-conf.xml >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extp= rj/g=3D >> > iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-wit= h-de=3D >> > pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-e= xamp=3D >> > les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name . >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking . >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/etc/hadoop >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/etc/hadoop >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/common/lib/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/common/lib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/common/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/common >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/hdfs >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/hdfs >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/hdfs/lib/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/hdfs/lib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/hdfs/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/hdfs >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/yarn/lib/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/yarn/lib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/yarn/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/yarn >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/mapreduce/lib/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/mapreduce/lib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/hadoop/mapreduce/* >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/mapreduce >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/myLib/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/myLib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extpr= j/gi=3D >> > raph/myGiraph/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/ex= tprj=3D >> > /giraph/myGiraph >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extpr= j/gi=3D >> > raph/myGiraph/lib/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/ex= tprj=3D >> > /giraph/myGiraph/lib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacit= y-sc=3D >> > heduler/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capa= city=3D >> > -scheduler >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionCont= ext =3D >> > for GiraphApplicationMaster launch container is populated. >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to= ASM >> > 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application appli= cati=3D >> > on_1415264041937_0005 >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submi= ssio=3D >> > n :application_1415264041937_0005 >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster c= onta=3D >> > iner request was submitted to ResourceManager for job: Giraph: org.apac= he.g=3D >> > iraph.examples.SimpleShortestPathsComputation >> > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph= .exa=3D >> > mples.SimpleShortestPathsComputation, Elapsed: 0,83 secs >> > 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_= 0005=3D >> > _000001, State: ACCEPTED, Containers used: 1 >> > 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.girap= h.ex=3D >> > amples.SimpleShortestPathsComputation reports FAILED state, diagnostics= sho=3D >> > w: Application application_1415264041937_0005 failed 2 times due to AM C= ont=3D >> > ainer for appattempt_1415264041937_0005_000002 exited with exitCode: 1= due=3D >> > to: Exception from container-launch: ExitCodeException exitCode=3D3D1: >> > ExitCodeException exitCode=3D3D1: >> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >> > at org.apache.hadoop.util.Shell.run(Shell.java:455) >> > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java= :702=3D >> > ) >> > at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.l= aunc=3D >> > hContainer(DefaultContainerExecutor.java:195) >> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.= Cont=3D >> > ainerLaunch.call(ContainerLaunch.java:300) >> > at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.= Cont=3D >> > ainerLaunch.call(ContainerLaunch.java:81) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor= .jav=3D >> > a:1145) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto= r.ja=3D >> > va:615) >> > at java.lang.Thread.run(Thread.java:744) >> > >> > >> > Container exited with a non-zero exit code 1 >> > .Failing this attempt.. Failing the application. >> > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distribu= ted =3D >> > cache directory for Giraph job. >> > 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apa= che.=3D >> > giraph.examples.SimpleShortestPathsComputation: FAILED, total running t= ime:=3D >> > 0 minutes, 3 seconds. >> > ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_141= 5264=3D >> > 041937_0005 >> > >> > As you can see there is this piece: >> > >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extp= rj/g=3D >> > iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-wit= h-de=3D >> > pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-e= xamp=3D >> > les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >> > >> > >> > But I continue to receive the error: >> > >> > Thanks, >> > Alessandro >> > >> > >> > Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh > > ailto:tripti@yahoo-inc.com>> ha scritto: >> > >> > For GiraphApplicationMaster not found, the simple thing that worked for= me =3D >> > was adding a -yj parameter to my command. >> > (hadoop jar {jar-file-path} {path to giraph runner} {computation-class}= {bu=3D >> > nch of parameters} -yj {jar-file-name}) >> > If u look at job logs, >> > Do u see a log statement defining "LIB JARS"? >> > >> > Sent from my iPhone >> > >> > On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" > > alenegro81@yahoo.it>> wrote: >> > >> > Hi, >> > yea my logs shows that error. I follow this example: >> > >> > http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4= C68=3D >> > 9.5060101@web.de%3E >> > >> > without success. >> > Then I try to add all jars I can from giraph adding the following lines= int=3D >> > o the hadoop-env.sh: >> > >> > export MYLIB=3D3D/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/ex= tprj/g=3D >> > iraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar >> > export HADOOP_CLASSPATH=3D3D$HADOOP_CLASSPATH:$MYLIB >> > >> > >> > when I run the application now I see this: >> > >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/hadoop/mapreduce >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop= -2.5=3D >> > .1/share/myLib/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/had= oop-=3D >> > 2.5.1/share/myLib >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extpr= j/gi=3D >> > raph/myGiraph/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/ex= tprj=3D >> > /giraph/myGiraph >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extpr= j/gi=3D >> > raph/myGiraph/lib/*.jar >> > 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/ex= tprj=3D >> > /giraph/myGiraph/lib >> > >> > >> > But again the same error. What I think is that there is no problem in t= he l=3D >> > ibrary but while is starting up. >> > I saw this piece in the logs: >> > >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionCont= ext =3D >> > for GiraphApplicationMaster launch container is populated. >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to= ASM >> > 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application appli= cati=3D >> > on_1415264041937_0005 >> > 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submi= ssio=3D >> > n :application_141526 >>=20 --Apple-Mail-BC7C39D7-1E69-422E-AE56-5745D70FA6F7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
Hi Das,
I mean as user you run Hadoop yarn resource manager.

Alessandro

Il giorno 20/nov/2014, alle ore 19:22, D Adams <dadamszx@gmail.com> ha scritto:

Alessandro,
     I'm not sure what you mean, should I create a new user on my system named 'yarn'? I'm new to both hadoop and giraph, so I'm not sure.

Tripti,
     I'll get those logs as soon as I can.

Thank you both,

V/r
Das

On Nov 20, 2014 3:13 AM, "Alessandro Negro" <alenegro81@yahoo.it> wrote:
Hi Tripti,
I agree that a more detailed error log could be useful.

Thanks,
Alessandro




Il giorno 19/nov/2014, alle ore 15:54, Tripti Singh <tripti@yahoo-inc.com> ha scritto:

> --_000_D092AB9C1CB42triptiyahooinccom_
> Content-Type: text/plain; charset="Windows-1252"
> Content-Transfer-Encoding: quoted-printable
>
> Hi Alessandro,
>
> I guess u r talking about the documentation on Giraph webpage which mention=
> s addition of this new option which is more or less mandatory for yarn base=
> d profiles.
> When I first ran the Giraph on yarn without the proper =96yj option, there =
> was no way I could figure out from the Application/Container logs that this=
> was the issue.
> I think, it=92ll be a good idea to have this message in the logs for easy d=
> ebugging.
>
> Thanks,
> Tripti.
>
>
> From: Alessandro Negro <alenegro81@yahoo.it<mailto:alenegro81@yahoo.it>>
> Reply-To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@gir=
> aph.apache.org<mailto:user@giraph.apache.org>>
> Date: Tuesday, November 18, 2014 at 4:31 PM
> To: "user@giraph.apache.org<mailto:user@giraph.apache.org>" <user@giraph.ap=
> ache.org<mailto:user@giraph.apache.org>>
> Subject: Re: Help with Giraph on Yarn
>
> Hi Eli,
> no I think that the help message is clear enough, generally when I read =93=
> jar=94 I always mean absolute path for jar, but this is a my personal misun=
> derstanding.
>
> Thanks,
> Alessandro
>
>
> Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <apache.mailbox@gmail.co=
> m<mailto:apache.mailbox@gmail.com>> ha scritto:
>
> Sounds like you got it figured out since last time I checked this list, sor=
> ry it was a pain. Feel free to drop a JIRA ticket if we can make help messa=
> ge etc. for -yj option more clear, there's lots to improve there.
>
> On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro <alenegro81@yahoo.it<mailt=
> o:alenegro81@yahoo.it>> wrote:
> Hi Tripti,
> finally I was able to run the test with success. It was an issue of permiss=
> ion since I was running as ale not as yarn.
> Let me say that now I=92m able to run graph examples on Yarn 2.5.1. This is=
> the final result:
>
> 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.=
> giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running ti=
> me: 0 minutes, 21 seconds.
>
>
> Many thanks for your support,
> Alessandro
>
> Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <tripti@yahoo-inc.com<m=
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> I don't know if u have access to this node. But if u do, u could check if t=
> he file is indeed there and u have access to it.
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:=
> alenegro81@yahoo.it>> wrote:
>
> You are right it works, but now I receive the following error:
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=
> rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-exam=
> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i=
> mpl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=
> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana=
> tion.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCD=
> ynamicStore
> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Coul=
> d not configure the containerlaunch context for GiraphYarnTasks.
> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
> ApplicationMaster.java:391)
> at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat=
> ionMaster.java:78)
> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b=
> uildContainerLaunchContext(GiraphApplicationMaster.java:522)
> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r=
> un(GiraphApplicationMaster.java:479)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
> a:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
> va:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoo=
> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_000=
> 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
> stem.java:1072)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
> stem.java:1064)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv=
> er.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi=
> leSystem.java:1064)
> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153=
> )
> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
> ApplicationMaster.java:387)
> ... 6 more
>
>
> That justify the other error I receive in the task:
> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask
>
> Thanks,
>
> Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <tripti@yahoo-inc.com<m=
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> Why r u adding two jars? Example jar ideally contains core library so every=
> thing should be available with just one example jar included
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:=
> alenegro81@yahoo.it>> wrote:
>
> Hi,
> now it seems better, I need to add:
>
> giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-exa=
> mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
>
> Now it seems that after a lot of cycle it fail with this error:
>
> Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask
>
> But in this case the error appear in task-3-stderr.log  not in gam-stderr.l=
> og where there is the following error:
>
> LF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=
> rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.=
> 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/Stati=
> cLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use=
> rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-exam=
> ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i=
> mpl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo=
> n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana=
> tion.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDy=
> namicStore
> Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Coul=
> d not configure the containerlaunch context for GiraphYarnTasks.
> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
> ApplicationMaster.java:391)
> at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat=
> ionMaster.java:78)
> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b=
> uildContainerLaunchContext(GiraphApplicationMaster.java:522)
> at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r=
> un(GiraphApplicationMaster.java:479)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
> a:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
> va:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoo=
> p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_000=
> 6/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
> stem.java:1072)
> at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy=
> stem.java:1064)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv=
> er.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi=
> leSystem.java:1064)
> at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153=
> )
> at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77)
> at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph=
> ApplicationMaster.java:387)
> ... 6 more
>
>
> Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tripti@yahoo-inc.com<m=
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> Hi,
> I remember seeing this thread, but since i was already deep into fixing thi=
> s issue in production, I didn't really wait for the patch, not even the mun=
> ge symbol as I was not even able to compile the yarn profile( sorry for bei=
> ng so ruthless to giraph :-( )
> I did later cross check with the patch and it was quite similar to my chang=
> e but sure it wouldn't work for all profiles(in fact some profiles may even=
> break due to Hadoop class incompatibility ) I can share those changes with=
> u once I get back to my computer but I don't think that could have caused =
> this problem.
> Just try the small changed I mentioned in my previous reply and let me know=
> if that helped.
>
> Thanks,
> Tripti
> Sent from my iPhone
>
> On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:=
> alenegro81@yahoo.it>> wrote:
>
> Hi Tripti,
> let me add one info, As you can see from this thread of email:
>
> http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3CC3F75A9=
> 8-BDE8-4205-971F-1F7241E6E83F@yahoo.it%3E
>
> I had some issue while compiling for Hadoop 2.5.1 could be that into the pr=
> ocess I loose something?
>
> Thanks again,
> Alessandro
>
>
>
> Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegro81@yahoo.i=
> t<mailto:alenegro81@yahoo.it>> ha scritto:
>
> Hi Tripti,
> yes I run in this way:
>
>
> $HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/tar=
> get/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.j=
> ar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortest=
> PathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubl=
> eVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.I=
> dWithValueTextOutputFormat -op /shortestpaths -w 1 -yj /Users/ale/extprj/gi=
> raph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dep=
> endencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-exampl=
> es-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
> And this is the complete trace:
>
> 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from SCDy=
> namicStore
> 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop =
> library for your platform... using builtin-java classes where applicable
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format speci=
> fied. Ensure your InputFormat does not require one.
> 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format spec=
> ified. Ensure your OutputFormat does not require one.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdfs://=
> hadoop-master:9000/shortestpaths
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client
> 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at /0.=
> 0.0.0:8032<http://0.0.0.0:8032/>
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for,=
> nodeId=3D10.61.137.28:59677<http://10.61.137.28:59677/>, nodeAddress 10.61=
> .137.28:8042<http://10.61.137.28:8042/>, nodeRackName /default-rack, nodeNu=
> mContainers 0
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size=
> to 1024MB.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: =
> application_1415264041937_0005
> 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id<http://mapr=
> ed.job.id/> is deprecated. Instead, use mapreduce.job.id<http://mapreduce.j=
> ob.id/>
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the a=
> pplication master
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPAT=
> H=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/comm=
> on/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/=
> hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/s=
> hare/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED=
> _HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/l=
> ib/*}
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources ::=
> hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_14152=
> 64041937_0005/giraph-conf.xml
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/g=
> iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-de=
> pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examp=
> les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking .
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/etc/hadoop
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/common/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/common/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/common/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/common
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/hdfs/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/hdfs/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/hdfs/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/hdfs
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/yarn/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/yarn/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/yarn/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/yarn
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/mapreduce/lib/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/mapreduce/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/hadoop/mapreduce/*
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/mapreduce
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/myLib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/myLib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi=
> raph/myGiraph/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj=
> /giraph/myGiraph
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi=
> raph/myGiraph/lib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj=
> /giraph/myGiraph/lib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacity-sc=
> heduler/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capacity=
> -scheduler
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext =
> for GiraphApplicationMaster launch container is populated.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application applicati=
> on_1415264041937_0005
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submissio=
> n :application_1415264041937_0005
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster conta=
> iner request was submitted to ResourceManager for job: Giraph: org.apache.g=
> iraph.examples.SimpleShortestPathsComputation
> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa=
> mples.SimpleShortestPathsComputation, Elapsed: 0,83 secs
> 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005=
> _000001, State: ACCEPTED, Containers used: 1
> 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex=
> amples.SimpleShortestPathsComputation reports FAILED state, diagnostics sho=
> w: Application application_1415264041937_0005 failed 2 times due to AM Cont=
> ainer for appattempt_1415264041937_0005_000002 exited with  exitCode: 1 due=
> to: Exception from container-launch: ExitCodeException exitCode=3D1:
> ExitCodeException exitCode=3D1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702=
> )
> at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc=
> hContainer(DefaultContainerExecutor.java:195)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont=
> ainerLaunch.call(ContainerLaunch.java:300)
> at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont=
> ainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav=
> a:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja=
> va:615)
> at java.lang.Thread.run(Thread.java:744)
>
>
> Container exited with a non-zero exit code 1
> .Failing this attempt.. Failing the application.
> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed =
> cache directory for Giraph job.
> 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.=
> giraph.examples.SimpleShortestPathsComputation: FAILED, total running time:=
> 0 minutes, 3 seconds.
> ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_1415264=
> 041937_0005
>
> As you can see there is this piece:
>
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/g=
> iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-de=
> pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examp=
> les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
>
>
> But I continue to receive the error:
>
> Thanks,
> Alessandro
>
>
> Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tripti@yahoo-inc.com<m=
> ailto:tripti@yahoo-inc.com>> ha scritto:
>
> For GiraphApplicationMaster not found, the simple thing that worked for me =
> was adding a -yj parameter to my command.
> (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} {bu=
> nch of parameters} -yj {jar-file-name})
> If u look at job logs,
> Do u see a log statement defining "LIB JARS"?
>
> Sent from my iPhone
>
> On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegro81@yahoo.it<mailto:=
> alenegro81@yahoo.it>> wrote:
>
> Hi,
> yea my logs shows that error. I follow this example:
>
> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C68=
> 9.5060101@web.de%3E
>
> without success.
> Then I try to add all jars I can from giraph adding the following lines int=
> o the hadoop-env.sh:
>
> export MYLIB=3D/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/g=
> iraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
> export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$MYLIB
>
>
> when I run the application now I see this:
>
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/hadoop/mapreduce
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5=
> .1/share/myLib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-=
> 2.5.1/share/myLib
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi=
> raph/myGiraph/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj=
> /giraph/myGiraph
> 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi=
> raph/myGiraph/lib/*.jar
> 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj=
> /giraph/myGiraph/lib
>
>
> But again the same error. What I think is that there is no problem in the l=
> ibrary but while is starting up.
> I saw this piece in the logs:
>
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext =
> for GiraphApplicationMaster launch container is populated.
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM
> 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application applicati=
> on_1415264041937_0005
> 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submissio=
> n :application_141526

--Apple-Mail-BC7C39D7-1E69-422E-AE56-5745D70FA6F7--