Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C088D10EE6 for ; Wed, 19 Nov 2014 14:58:03 +0000 (UTC) Received: (qmail 62263 invoked by uid 500); 19 Nov 2014 14:58:03 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 62226 invoked by uid 500); 19 Nov 2014 14:58:03 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 62216 invoked by uid 99); 19 Nov 2014 14:58:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Nov 2014 14:58:03 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 98.139.253.104 is neither permitted nor denied by domain of tripti@yahoo-inc.com) Received: from [98.139.253.104] (HELO mrout1-b.corp.bf1.yahoo.com) (98.139.253.104) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Nov 2014 14:57:30 +0000 Received: from GQ1-EX10-CAHT07.y.corp.yahoo.com (gq1-ex10-caht07.corp.gq1.yahoo.com [10.73.118.86]) by mrout1-b.corp.bf1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id sAJErupF085913 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 19 Nov 2014 06:55:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1416408929; bh=IdZmv9Gb9alxXl0rP0F3Y3orShCnZg8D3J9/zW+is94=; h=From:To:Subject:Date:References:In-Reply-To; b=lfAIGhQ4CTi6cgrW2eQiGFKIcxJfgCVpe4Tsx5B2Eg6AZrx8sSBC4OwbA59nQ5lXy /6aXT7VeZ/9i28GUBtCEbdOCwvelIf12yjxXC257Ojl4SSkcWgPrOxzz12hQc1LjSE AIhAPS726Nhu6qFGSeUZOza9nHB6Bia43WQpQs5E= Received: from GQ1-EX10-DRMB03.y.corp.yahoo.com ([fe80::c153:a192:3d57:5949]) by GQ1-EX10-CAHT07.y.corp.yahoo.com ([::1]) with mapi id 14.03.0195.001; Wed, 19 Nov 2014 06:54:28 -0800 From: Tripti Singh To: "user@giraph.apache.org" Subject: Re: Help with Giraph on Yarn Thread-Topic: Help with Giraph on Yarn Thread-Index: AQHP+RZzSlwoMDrXsk6bywNyBIzOWZxSxc0AgAADKgCAAAEaAP//fI7ggACINYCAAAJRAIAACnMAgAAl2oCAAPtrAP//2x2A//988nkAEY14AAABD/KA//990J2AAI57gP//jCzSgACOz4CAABoFgIADOYMAgAzqPICAAoMvgIACL3mA Date: Wed, 19 Nov 2014 14:54:27 +0000 Message-ID: References: <77A73EB3-7967-4AB4-B5E6-191B46898119@yahoo.it> <0B32BE05-B6B6-4FEF-B2E8-8B99A45F4A83@yahoo.it> <41384542-C90C-4573-992E-CDD5E29B5803@yahoo-inc.com> <536AC63F-4C87-4071-A1D9-D867DCC6769B@yahoo.it> <045D183C-ED29-40F9-A0B5-A6A57110A919@yahoo.it> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.5.141003 x-originating-ip: [10.66.183.146] Content-Type: multipart/alternative; boundary="_000_D092AB9C1CB42triptiyahooinccom_" MIME-Version: 1.0 X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 408929001 X-Virus-Checked: Checked by ClamAV on apache.org --_000_D092AB9C1CB42triptiyahooinccom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Alessandro, I guess u r talking about the documentation on Giraph webpage which mention= s addition of this new option which is more or less mandatory for yarn base= d profiles. When I first ran the Giraph on yarn without the proper =96yj option, there = was no way I could figure out from the Application/Container logs that this= was the issue. I think, it=92ll be a good idea to have this message in the logs for easy d= ebugging. Thanks, Tripti. From: Alessandro Negro > Reply-To: "user@giraph.apache.org" > Date: Tuesday, November 18, 2014 at 4:31 PM To: "user@giraph.apache.org" > Subject: Re: Help with Giraph on Yarn Hi Eli, no I think that the help message is clear enough, generally when I read =93= jar=94 I always mean absolute path for jar, but this is a my personal misun= derstanding. Thanks, Alessandro Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman > ha scritto: Sounds like you got it figured out since last time I checked this list, sor= ry it was a pain. Feel free to drop a JIRA ticket if we can make help messa= ge etc. for -yj option more clear, there's lots to improve there. On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro > wrote: Hi Tripti, finally I was able to run the test with success. It was an issue of permiss= ion since I was running as ale not as yarn. Let me say that now I=92m able to run graph examples on Yarn 2.5.1. This is= the final result: 14/11/08 16:24:00 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.= giraph.examples.SimpleShortestPathsComputation: SUCCEEDED, total running ti= me: 0 minutes, 21 seconds. Many thanks for your support, Alessandro Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh > ha scritto: I don't know if u have access to this node. But if u do, u could check if t= he file is indeed there and u have access to it. Sent from my iPhone On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" > wrote: You are right it works, but now I receive the following error: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use= rcache/ale/appcache/application_1415264041937_0009/filecache/10/giraph-exam= ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i= mpl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo= n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= tion. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 13:15:37.120 java[10158:1803] Unable to load realm info from SCD= ynamicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Coul= d not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= ApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= ionMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= uildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= un(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoo= p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_000= 9/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= stem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= stem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= er.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= leSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= ) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= ApplicationMaster.java:387) ... 6 more That justify the other error I receive in the task: Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask Thanks, Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh > ha scritto: Why r u adding two jars? Example jar ideally contains core library so every= thing should be available with just one example jar included Sent from my iPhone On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" > wrote: Hi, now it seems better, I need to add: giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-exa= mples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar Now it seems that after a lot of cycle it fail with this error: Could not find or load main class org.apache.giraph.yarn.GiraphYarnTask But in this case the error appear in task-3-stderr.log not in gam-stderr.l= og where there is the following error: LF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use= rcache/ale/appcache/application_1415264041937_0006/filecache/12/giraph-1.1.= 0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/Stati= cLoggerBinder.class] SLF4J: Found binding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/use= rcache/ale/appcache/application_1415264041937_0006/filecache/10/giraph-exam= ples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/i= mpl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/commo= n/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explana= tion. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2014-11-06 11:31:25.186 java[8162:1803] Unable to load realm info from SCDy= namicStore Exception in thread "pool-4-thread-1" java.lang.IllegalStateException: Coul= d not configure the containerlaunch context for GiraphYarnTasks. at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= ApplicationMaster.java:391) at org.apache.giraph.yarn.GiraphApplicationMaster.access$500(GiraphApplicat= ionMaster.java:78) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.b= uildContainerLaunchContext(GiraphApplicationMaster.java:522) at org.apache.giraph.yarn.GiraphApplicationMaster$LaunchContainerRunnable.r= un(GiraphApplicationMaster.java:479) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hadoo= p-master:9000/user/yarn/giraph_yarn_jar_cache/application_1415264041937_000= 6/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= stem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSy= stem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolv= er.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFi= leSystem.java:1064) at org.apache.giraph.yarn.YarnUtils.addFileToResourceMap(YarnUtils.java:153= ) at org.apache.giraph.yarn.YarnUtils.addFsResourcesToMap(YarnUtils.java:77) at org.apache.giraph.yarn.GiraphApplicationMaster.getTaskResourceMap(Giraph= ApplicationMaster.java:387) ... 6 more Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh > ha scritto: Hi, I remember seeing this thread, but since i was already deep into fixing thi= s issue in production, I didn't really wait for the patch, not even the mun= ge symbol as I was not even able to compile the yarn profile( sorry for bei= ng so ruthless to giraph :-( ) I did later cross check with the patch and it was quite similar to my chang= e but sure it wouldn't work for all profiles(in fact some profiles may even= break due to Hadoop class incompatibility ) I can share those changes with= u once I get back to my computer but I don't think that could have caused = this problem. Just try the small changed I mentioned in my previous reply and let me know= if that helped. Thanks, Tripti Sent from my iPhone On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" > wrote: Hi Tripti, let me add one info, As you can see from this thread of email: http://mail-archives.apache.org/mod_mbox/giraph-user/201410.mbox/%3CC3F75A9= 8-BDE8-4205-971F-1F7241E6E83F@yahoo.it%3E I had some issue while compiling for Hadoop 2.5.1 could be that into the pr= ocess I loose something? Thanks again, Alessandro Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro > ha scritto: Hi Tripti, yes I run in this way: $HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/tar= get/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.j= ar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortest= PathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubl= eVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.I= dWithValueTextOutputFormat -op /shortestpaths -w 1 -yj /Users/ale/extprj/gi= raph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dep= endencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-exampl= es-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar And this is the complete trace: 2014-11-06 10:06:05.638 java[7778:1903] Unable to load realm info from SCDy= namicStore 14/11/06 10:06:05 WARN util.NativeCodeLoader: Unable to load native-hadoop = library for your platform... using builtin-java classes where applicable 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge input format speci= fied. Ensure your InputFormat does not require one. 14/11/06 10:06:06 INFO utils.ConfigurationUtils: No edge output format spec= ified. Ensure your OutputFormat does not require one. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Final output path is: hdfs://= hadoop-master:9000/shortestpaths 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Running Client 14/11/06 10:06:06 INFO client.RMProxy: Connecting to ResourceManager at /0.= 0.0.0:8032 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got node report from ASM for,= nodeId=3D10.61.137.28:59677, nodeAddress 10.61= .137.28:8042, nodeRackName /default-rack, nodeNu= mContainers 0 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Defaulting per-task heap size= to 1024MB. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Obtained new Application ID: = application_1415264041937_0005 14/11/06 10:06:06 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Set the environment for the a= pplication master 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPAT= H=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/comm= on/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/= hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/s= hare/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED= _HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/l= ib/*} 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... 14/11/06 10:06:06 INFO yarn.YarnUtils: Registered file in LocalResources ::= hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_14152= 64041937_0005/giraph-conf.xml 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/g= iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-de= pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examp= les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name . 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking . 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name null 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking null 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/etc/hadoop 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/etc/hadoop 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/common/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/common/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/common/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/common 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/yarn/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/yarn/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/yarn/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/yarn 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/mapreduce/lib/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/mapreduce/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/mapreduce/* 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi= raph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj= /giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi= raph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj= /giraph/myGiraph/lib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /contrib/capacity-sc= heduler/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /contrib/capacity= -scheduler 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext = for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application applicati= on_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submissio= n :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster conta= iner request was submitted to ResourceManager for job: Giraph: org.apache.g= iraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 0,83 secs 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005= _000001, State: ACCEPTED, Containers used: 1 14/11/06 10:06:10 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex= amples.SimpleShortestPathsComputation reports FAILED state, diagnostics sho= w: Application application_1415264041937_0005 failed 2 times due to AM Cont= ainer for appattempt_1415264041937_0005_000002 exited with exitCode: 1 due= to: Exception from container-launch: ExitCodeException exitCode=3D1: ExitCodeException exitCode=3D1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702= ) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc= hContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed = cache directory for Giraph job. 14/11/06 10:06:10 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.= giraph.examples.SimpleShortestPathsComputation: FAILED, total running time:= 0 minutes, 3 seconds. ale:~/extprj/giraph/myGiraph />yarn logs -applicationId application_1415264= 041937_0005 As you can see there is this piece: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/g= iraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-de= pendencies.jar,/Users/ale/extprj/giraph/giraph-examples/target/giraph-examp= les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar But I continue to receive the error: Thanks, Alessandro Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh > ha scritto: For GiraphApplicationMaster not found, the simple thing that worked for me = was adding a -yj parameter to my command. (hadoop jar {jar-file-path} {path to giraph runner} {computation-class} {bu= nch of parameters} -yj {jar-file-name}) If u look at job logs, Do u see a log statement defining "LIB JARS"? Sent from my iPhone On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" > wrote: Hi, yea my logs shows that error. I follow this example: http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C53F4C68= 9.5060101@web.de%3E without success. Then I try to add all jars I can from giraph adding the following lines int= o the hadoop-env.sh: export MYLIB=3D/opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/g= iraph/myGiraph/*.jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar export HADOOP_CLASSPATH=3D$HADOOP_CLASSPATH:$MYLIB when I run the application now I see this: 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/mapreduce 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/myLib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/myLib 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi= raph/myGiraph/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj= /giraph/myGiraph 14/11/06 10:06:06 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/gi= raph/myGiraph/lib/*.jar 14/11/06 10:06:06 INFO yarn.YarnUtils: base path checking /Users/ale/extprj= /giraph/myGiraph/lib But again the same error. What I think is that there is no problem in the l= ibrary but while is starting up. I saw this piece in the logs: 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext = for GiraphApplicationMaster launch container is populated. 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/06 10:06:06 INFO impl.YarnClientImpl: Submitted application applicati= on_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: Got new appId after submissio= n :application_1415264041937_0005 14/11/06 10:06:06 INFO yarn.GiraphYarnClient: GiraphApplicationMaster conta= iner request was submitted to ResourceManager for job: Giraph: org.apache.g= iraph.examples.SimpleShortestPathsComputation 14/11/06 10:06:07 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 0,83 secs That show that it is able to load GiraphApplicationMaster but fail during t= he startup. Where can I find more logs? Thanks, Alessandro Il giorno 06/nov/2014, alle ore 06:56, Tripti Singh > ha scritto: Can u run the following command to find container logs and post it here: yarn logs -applicationId application_1415203231170_0002 If u find this error: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationM= aster It may just be due to non-availablity of the yarn jars Thanks, Tripti Tech Yahoo From: Jing Fan > Reply-To: "user@giraph.apache.org" > Date: Thursday, November 6, 2014 at 1:56 AM To: "user@giraph.apache.org" > Subject: Re: Help with Giraph on Yarn Then do you know to solve this error? 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex= amples.SimpleShortestPathsComputation reports FAILED state, diagnostics sho= w: Application application_1415203231170_0002 failed 2 times due to AM Cont= ainer for appattempt_1415203231170_0002_000002 exited with exitCode: 1 due= to: Exception from container-launch. Container id: container_1415203231170_0002_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=3D1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702= ) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc= hContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:745) On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negro > wrote: Hi, in the mean time I found the reason why it hangs for long time here: base path checking . It scan all the directory tree starting from current directory, if you run = it into your home (and you have a lot of files and directories) it takes ag= es to complete. I continue to got the same error. Regards, Alessandro Il giorno 05/nov/2014, alle ore 18:33, Jing Fan > ha scritto: I run with: hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop= -2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner or= g.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.gir= aph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/t= iny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat= -op /user/jing/shortestpaths -w 1 -yj giraph-core/target/giraph-1.1.0-SNAP= SHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro > wrote: This is my complete logs with the command I run: ale:~ />$HADOOP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-exam= ples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-depend= encies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.Simple= ShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFl= oatDoubleVertexInputFormat -vip /tiny_graph.txt -vof org.apache.giraph.io.f= ormats.IdWithValueTextOutputFormat -op /shortestpaths -w 1 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from SCDy= namicStore 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop = library for your platform... using builtin-java classes where applicable 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format speci= fied. Ensure your InputFormat does not require one. 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format spec= ified. Ensure your OutputFormat does not require one. 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: hdfs://= hadoop-master:9000/shortestpaths 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at /0.= 0.0.0:8032 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for,= nodeId=3D192.168.0.105:50658, nodeAddress 192= .168.0.105:8042, nodeRackName /default-rack, no= deNumContainers 0 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size= to 1024MB. 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: = application_1415189069846_0004 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the a= pplication master 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPAT= H=3D${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/comm= on/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/= hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/s= hare/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED= _HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/l= ib/*} 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources ::= hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_14151= 89069846_0004/giraph-conf.xml 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/etc/hadoop 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/etc/hadoop 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/common/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/common/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/common/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/common 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/hdfs/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/hdfs 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/yarn/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/yarn/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/yarn/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/yarn 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/mapreduce/lib/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/mapreduce/lib 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5= .1/share/hadoop/mapreduce/* 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-= 2.5.1/share/hadoop/mapreduce 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name /contrib/capacity-sc= heduler/*.jar 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking /contrib/capacity= -scheduler 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext = for GiraphApplicationMaster launch container is populated. 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application applicati= on_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submissio= n :application_1415189069846_0004 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster conta= iner request was submitted to ResourceManager for job: Giraph: org.apache.g= iraph.examples.SimpleShortestPathsComputation 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 0,82 secs 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004= _000001, State: ACCEPTED, Containers used: 1 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex= amples.SimpleShortestPathsComputation reports FAILED state, diagnostics sho= w: Application application_1415189069846_0004 failed 2 times due to AM Cont= ainer for appattempt_1415189069846_0004_000002 exited with exitCode: 1 due= to: Exception from container-launch: ExitCodeException exitCode=3D1: ExitCodeException exitCode=3D1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702= ) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc= hContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed = cache directory for Giraph job. 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.= giraph.examples.SimpleShortestPathsComputation: FAILED, total running time:= 0 minutes, 3 seconds. Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh > ha scritto: I faced the similar problem where the workflow just hangs. Can u see in ur logs if there is a task with id 0. I made a local fix and I could give u the patch if u see a similar problem = and ur workflow is unable to find the zookeeper manager file and repeats th= e following message: For task xyz, got file 'null' Thanks, Tripti Sent from my iPhone On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" > wrote: Hi, have you seen the hadoop logs? Could you write here how are you running the command? Thanks, Alessandro Il giorno 05/nov/2014, alle ore 18:04, Jing Fan > ha scritto: Yeah, I add -yj option. The result is the same. Still error here. On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro > wrote: Hi Jing, I=92m facing the same problem. Do you check the hadoop logs? In my case I g= ot: Could not find or load main class org.apache.giraph.yarn.GiraphApplicationM= aster Moreover I don=92t know why my application hang for a long time here: 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . And then continue failing with your error into the output and the other int= o the hadoop logs. Did you test adding this? -yj /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-h= adoop-2.5.1-jar-with-dependencies.jar Thanks, Alessandro Il giorno 05/nov/2014, alle ore 17:32, Jing Fan > ha scritto: Hi, I install the Giraph with Yarn. But when I run the program with following command: hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoo= p-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner o= rg.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.gi= raph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/j/tin= y_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -= op /user/j/shortestpaths -w 1 The program exited with error: 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 0.83 secs 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002= _000001, State: ACCEPTED, Containers used: 0 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 4.84 secs 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002= _000001, State: ACCEPTED, Containers used: 0 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 8.85 secs 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002= _000002, State: ACCEPTED, Containers used: 0 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimpleShortestPathsComputation, Elapsed: 12.86 secs 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: appattempt_1415203231170_0002= _000002, State: ACCEPTED, Containers used: 0 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex= amples.SimpleShortestPathsComputation reports FAILED state, diagnostics sho= w: Application application_1415203231170_0002 failed 2 times due to AM Cont= ainer for appattempt_1415203231170_0002_000002 exited with exitCode: 1 due= to: Exception from container-launch. Container id: container_1415203231170_0002_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=3D1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702= ) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launc= hContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Cont= ainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja= va:615) at java.lang.Thread.run(Thread.java:745) Anyone can help? --_000_D092AB9C1CB42triptiyahooinccom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <872E3E4EABAE764E9212748F8CE10024@yforest.corp.yahoo.com> Content-Transfer-Encoding: quoted-printable
Hi Alessandro,

I guess u r talking about the documentation on Giraph webpage which me= ntions addition of this new option which is more or less mandatory for yarn= based profiles.
When I first ran the Giraph on yarn without the proper =96yj option, t= here was no way I could figure out from the Application/Container logs that= this was the issue.
I think, it=92ll be a good idea to have this message in the logs for e= asy debugging.

Thanks,
Tripti.


From: Alessandro Negro <alenegro81@yahoo.it>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Tuesday, November 18, 2014 at= 4:31 PM
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Help with Giraph on Ya= rn

Hi Eli,
no I think that the help message is clear enough, generally when I rea= d =93jar=94 I always mean absolute path for jar, but this is a my personal = misunderstanding.

Thanks,
Alessandro


Il giorno 16/nov/2014, alle ore 21:39, Eli Reisman <apache.mailbox@gmail.com> ha scritto:
Sounds like you got it figured out since last time I check= ed this list, sorry it was a pain. Feel free to drop a JIRA ticket if we ca= n make help message etc. for -yj option more clear, there's lots to improve= there.

On Sat, Nov 8, 2014 at 7:26 AM, Alessandro Negro= <alenegro81@yah= oo.it> wrote:
Hi Tripti,
finally I was able to run the test with success. It was an issue of pe= rmission since I was running as ale not as yarn.
Let me say that now I=92m able to run graph examples on Yarn 2.5.1. Th= is is the final result:

14/11/08 16:24:0= 0 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.= SimpleShortestPathsComputation: SUCCEEDED, total running time: 0 minutes, 2= 1 seconds.


Many thanks for your support,
Alessandro

Il giorno 06/nov/2014, alle ore 15:16, Tripti Singh <tripti@yahoo-inc.com> ha= scritto:

I don't know if u have access to this node. But if u do, u could check= if the file is indeed there and u have access to it.

Sent from my iPhone

On 06-Nov-2014, at 6:12 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrot= e:

You are right it works, but now I receive the following error:

SLF4J: Class pat= h contains multiple SLF4J bindings.
SLF4J: Found bin= ding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appca= che/application_1415264041937_0009/filecache/10/giraph-examples-1.1.0-SNAPS= HOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLogger= Binder.class]
SLF4J: Found bin= ding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4= j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual bi= nding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-11-06 13:15= :37.120 java[10158:1803] Unable to load realm info from SCDynamicStore
Exception in thr= ead "pool-4-thread-1" java.lang.IllegalStateException: Could not = configure the containerlaunch context for GiraphYarnTasks.
at org.apache.giraph.yarn.GiraphApplicationMast= er.getTaskResourceMap(GiraphApplicationMaster.java:391)
at org.apache.giraph.yarn.GiraphApplicationMast= er.access$500(GiraphApplicationMaster.java:78)
at org.apache.giraph.yarn.GiraphApplicationMast= er$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMas= ter.java:522)
at org.apache.giraph.yarn.GiraphApplicationMast= er$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
at java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at org.apache.hadoop.hdfs.DistributedFileSystem= $17.doCall(DistributedFileSystem.java:1072)
at org.apache.hadoop.hdfs.DistributedFileSystem= $17.doCall(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystemLinkResolver.= resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem= .getFileStatus(DistributedFileSystem.java:1064)
at org.apache.giraph.yarn.YarnUtils.addFileToRe= sourceMap(YarnUtils.java:153)
at org.apache.giraph.yarn.YarnUtils.addFsResour= cesToMap(YarnUtils.java:77)
at org.apache.giraph.yarn.GiraphApplicationMast= er.getTaskResourceMap(GiraphApplicationMaster.java:387)
... 6 more


That justify the other error I receive in the task:
Could not find o= r load main class org.apache.giraph.yarn.GiraphYarnTask

Thanks,

Il giorno 06/nov/2014, alle ore 13:07, Tripti Singh <tripti@yahoo-inc.com> ha= scritto:

Why r u adding two jars? Example jar ideally contains core library so = everything should be available with just one example jar included

Sent from my iPhone

On 06-Nov-2014, at 4:33 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrot= e:

Hi,
now it seems better, I need to add: 

giraph-1.1.0-SNA= PSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,giraph-examples-1.1.0-SNAP= SHOT-for-hadoop-2.5.1-jar-with-dependencies.jar


Now it seems that after a lot of cycle it fail with this error:

Could not find o= r load main class org.apache.giraph.yarn.GiraphYarnTask

But in this case the error appear in task-3-stderr.log  not in&nb= sp;gam-stderr.log where there is the following error:

LF4J: Class path= contains multiple SLF4J bindings.
SLF4J: Found bin= ding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appca= che/application_1415264041937_0006/filecache/12/giraph-1.1.0-SNAPSHOT-for-h= adoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.cl= ass]
SLF4J: Found bin= ding in [jar:file:/private/tmp/hadoop-yarn/nm-local-dir/usercache/ale/appca= che/application_1415264041937_0006/filecache/10/giraph-examples-1.1.0-SNAPS= HOT-for-hadoop-2.5.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLogger= Binder.class]
SLF4J: Found bin= ding in [jar:file:/opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/slf4j-log4= j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual bi= nding is of type [org.slf4j.impl.Log4jLoggerFactory]
2014-11-06 11:31= :25.186 java[8162:1803] Unable to load realm info from SCDynamicStore
Exception in thr= ead "pool-4-thread-1" java.lang.IllegalStateException: Could not = configure the containerlaunch context for GiraphYarnTasks.
at org.apache.giraph.yarn.GiraphApplicationMast= er.getTaskResourceMap(GiraphApplicationMaster.java:391)
at org.apache.giraph.yarn.GiraphApplicationMast= er.access$500(GiraphApplicationMaster.java:78)
at org.apache.giraph.yarn.GiraphApplicationMast= er$LaunchContainerRunnable.buildContainerLaunchContext(GiraphApplicationMas= ter.java:522)
at org.apache.giraph.yarn.GiraphApplicationMast= er$LaunchContainerRunnable.run(GiraphApplicationMaster.java:479)
at java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
at org.apache.hadoop.hdfs.DistributedFileSystem= $17.doCall(DistributedFileSystem.java:1072)
at org.apache.hadoop.hdfs.DistributedFileSystem= $17.doCall(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystemLinkResolver.= resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem= .getFileStatus(DistributedFileSystem.java:1064)
at org.apache.giraph.yarn.YarnUtils.addFileToRe= sourceMap(YarnUtils.java:153)
at org.apache.giraph.yarn.YarnUtils.addFsResour= cesToMap(YarnUtils.java:77)
at org.apache.giraph.yarn.GiraphApplicationMast= er.getTaskResourceMap(GiraphApplicationMaster.java:387)
... 6 more


Il giorno 06/nov/2014, alle ore 11:32, Tripti Singh <tripti@yahoo-inc.com> ha= scritto:

Hi,
I remember seeing this thread, but since i was already deep into fixin= g this issue in production, I didn't really wait for the patch, not even th= e munge symbol as I was not even able to compile the yarn profile( sorry fo= r being so ruthless to giraph :-( )
I did later cross check with the patch and it was quite similar to my chang= e but sure it wouldn't work for all profiles(in fact some profiles may even= break due to Hadoop class incompatibility ) I can share those changes with= u once I get back to my computer but I don't think that could have caused this problem.
Just try the small changed I mentioned in my previous reply and let me= know if that helped.

Thanks,
Tripti 
Sent from my iPhone

On 06-Nov-2014, at 3:52 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrot= e:

Hi Tripti,
let me add one info, As you can see from this thread of email: 


I had some issue while compiling for Hadoop 2.5.1 could be that into t= he process I loose something?

Thanks again,
Alessandro



Il giorno 06/nov/2014, alle ore 10:47, Alessandro Negro <alenegro81@yahoo.it> = ha scritto:

Hi Tripti,
yes I run in this way:


$HADOOP_INSTALL/= bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/giraph-examp= les-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar org.apache.gi= raph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputForm= at -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOu= tputFormat -op /shortestpaths -w 1 -yj /Users/ale/extprj/giraph/giraph-core= /target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/U= sers/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHO= T-for-hadoop-2.5.1-jar-with-dependencies.jar

And this is the complete trace:

2014-11-06 10:06= :05.638 java[7778:1903] Unable to load realm info from SCDynamicStore
14/11/06 10:06:0= 5 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your= platform... using builtin-java classes where applicable
14/11/06 10:06:0= 6 INFO utils.ConfigurationUtils: No edge input format specified. Ensure you= r InputFormat does not require one.
14/11/06 10:06:0= 6 INFO utils.ConfigurationUtils: No edge output format specified. Ensure yo= ur OutputFormat does not require one.
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Running Client
14/11/06 10:06:0= 6 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=3D10.61.137.28:59677, = nodeAddress 10.61.137.28:8042, nodeRackName /default-rack, nodeNumContainers 0
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415= 264041937_0005
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Set the environment for the application maste= r
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=3D${CLASSPATH}= :./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COM= MON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$H= ADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn= /*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hado= op/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_141= 5264041937_0005/giraph-conf.xml
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-cor= e/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/= Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSH= OT-for-hadoop-2.5.1-jar-with-dependencies.jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name .
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking .
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name null
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking null
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop=
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= common/lib/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/common/lib
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= common/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/common
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs/lib/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs/lib
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= yarn/lib/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/yarn/lib
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= yarn/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/yarn
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= mapreduce/lib/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/mapreduce/lib
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= mapreduce/*
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/mapreduce
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name 
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*= .jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLi= b
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.= jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph=
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/li= b/*.jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph= /lib
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplic= ationMaster launch container is populated.
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/06 10:06:0= 6 INFO impl.YarnClientImpl: Submitted application application_1415264041937= _0005
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1= 415264041937_0005
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was= submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.S= impleShortestPathsComputation
14/11/06 10:06:0= 7 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShor= testPathsComputation, Elapsed: 0,83 secs
14/11/06 10:06:0= 7 INFO yarn.GiraphYarnClient: appattempt_1415264041937_0005_000001, State: = ACCEPTED, Containers used: 1
14/11/06 10:06:1= 0 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleSho= rtestPathsComputation reports FAILED state, diagnostics show: Application a= pplication_1415264041937_0005 failed 2 times due to AM Container for appattempt_1415264041937_0005_000002 exite= d with  exitCode: 1 due to: Exception from container-launch: ExitCodeE= xception exitCode=3D1: 
ExitCodeExceptio= n exitCode=3D1: 
at org.apache.hadoop.util.Shell.runCommand(Shel= l.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:= 455)
at org.apache.hadoop.util.Shell$ShellCommandExe= cutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.De= faultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTa= sk.java:262)
at java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
=
=
Container exited= with a non-zero exit code 1
.Failing this at= tempt.. Failing the application.
14/11/06 10:06:1= 0 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory = for Giraph job.
14/11/06 10:06:1= 0 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.= SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 se= conds.
ale:~/extprj/gir= aph/myGiraph />yarn logs -applicationId application_1415264041937_0005

As you can see there is this piece:

14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: LIB JARS :/Users/ale/extprj/giraph/giraph-cor= e/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar,/= Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSH= OT-for-hadoop-2.5.1-jar-with-dependencies.jar


But I continue to receive the error:

Thanks,
Alessandro


Il giorno 06/nov/2014, alle ore 10:25, Tripti Singh <tripti@yahoo-inc.com> ha= scritto:

For GiraphApplicationMaster not found, the simple thing that worked fo= r me was adding a -yj parameter to my command. 
(hadoop jar {jar-file-path} {path to giraph runner} {computation-class= } {bunch of parameters} -yj {jar-file-name})
If u look at job logs,
Do u see a log statement defining "LIB JARS"?

Sent from my iPhone

On 06-Nov-2014, at 2:45 pm, "Alessandro Negro" <alenegro81@yahoo.it> wrot= e:

Hi,
yea my logs shows that error. I follow this example:


without success.
Then I try to add all jars I can from giraph adding the following line= s into the hadoop-env.sh:

export MYLIB=3D/= opt/yarn/hadoop-2.5.1/share/myLib/*.jar:/Users/ale/extprj/giraph/myGiraph/*= .jar:/Users/ale/extprj/giraph/myGiraph/lib/*.jar
export HADOOP_CL= ASSPATH=3D$HADOOP_CLASSPATH:$MYLIB


when I run the application now I see this:

14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/mapreduce
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name 
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/myLib/*= .jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/myLi= b
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/*.= jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph=
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: Class path name /Users/ale/extprj/giraph/myGiraph/li= b/*.jar
14/11/06 10:06:0= 6 INFO yarn.YarnUtils: base path checking /Users/ale/extprj/giraph/myGiraph= /lib


But again the same error. What I think is that there is no problem in = the library but while is starting up.
I saw this piece in the logs:

14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplic= ationMaster launch container is populated.
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/06 10:06:0= 6 INFO impl.YarnClientImpl: Submitted application application_1415264041937= _0005
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1= 415264041937_0005
14/11/06 10:06:0= 6 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was= submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.S= impleShortestPathsComputation
14/11/06 10:06:0= 7 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShor= testPathsComputation, Elapsed: 0,83 secs


That show that it is able to load GiraphApplicationMaster but fail dur= ing the startup.
Where can I find more logs?

Thanks,
Alessandro
Can u run the = following command to find container logs and post it here:

yarn l= ogs -applicationId application_1415203231170_0002

If u find this error:
Could not find or load main class org.apache.giraph.yarn.GiraphApplica=
tionMaster
It may just be due to non-availablity of the yarn=
 jars
Thanks,
Tripti
Tech Yahoo

From: Jing Fan <fanjing09@gmail.com>
Reply-To: "user@giraph.apache.org" &= lt;user@giraph.= apache.org>
Date: Thursday, November 6, 2014 at= 1:56 AM
To: "user@giraph.apache.org" <user@giraph.apache= .org>
Subject: Re: Help with Giraph on Ya= rn

Then do you know to solve this error?

14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.gira= ph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostic= s show: Application application_1415203231170_0002 failed 2 times due to AM= Container for appattempt_1415203231170_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch.
Container id: container_1415203231170_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=3D1: 
at org.apache.hadoop.util.= Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.= Shell.run(Shell.java:455)
at org.apache.hadoop.util.= Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.= server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContaine= rExecutor.java:196)
at org.apache.hadoop.yarn.= server.nodemanager.containermanager.launcher.ContainerLaunch.call(Container= Launch.java:299)
at org.apache.hadoop.yarn.= server.nodemanager.containermanager.launcher.ContainerLaunch.call(Container= Launch.java:81)
at java.util.concurrent.Fu= tureTask.run(FutureTask.java:262)
at java.util.concurrent.Th= readPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Th= read.java:745)


On Wed, Nov 5, 2014 at 12:10 PM, Alessandro Negr= o <alenegro81@yah= oo.it> wrote:
Hi,
in the mean time I found the reason why it hangs for long time here:

base path checki= ng .


It scan all the directory tree starting from current directory, if you= run it into your home (and you have a lot of files and directories) it tak= es ages to complete.

I continue to got the same error.

Regards,
Alessandro

Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjing09@gmail.com> ha scrit= to:

I run with:

hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-h= adoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunn= er org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apach= e.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/jing/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithVal= ueTextOutputFormat -op /user/jing/shortestpaths -w 1 -yj giraph-core/target= /giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar<= br>

On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negr= o <alenegro81@yah= oo.it> wrote:
This is my complete logs with the comma= nd I run:


ale:~ />$HADO= OP_INSTALL/bin/hadoop jar /Users/ale/extprj/giraph/giraph-examples/target/g= iraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar or= g.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPaths= Computation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputForm= at -vip /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOu= tputFormat -op /shortestpaths -w 1
2014-11-05 13:39= :30.235 java[3147:1903] Unable to load realm info from SCDynamicStore
14/11/05 13:39:3= 0 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your= platform... using builtin-java classes where applicable
14/11/05 13:39:3= 0 INFO utils.ConfigurationUtils: No edge input format specified. Ensure you= r InputFormat does not require one.
14/11/05 13:39:3= 0 INFO utils.ConfigurationUtils: No edge output format specified. Ensure yo= ur OutputFormat does not require one.
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Final output path is: hdfs://hadoop-master:9000/shortestpaths
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Running Client
14/11/05 13:39:3= 1 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Got node report from ASM for, nodeId=3D192.168.0.105:50658= , nodeAddress 192.168.0.105:8042= , nodeRackName /default-rack, nodeNumContainers 0
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB.
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Obtained new Application ID: application_1415= 189069846_0004
14/11/05 13:39:3= 1 INFO Configuration.deprecation: mapred.job.id is de= precated. Instead, use mapreduce.job.id=
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Set the environment for the application maste= r
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: Environment for AM :{CLASSPATH=3D${CLASSPATH}= :./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COM= MON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$H= ADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn= /*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hado= op/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*}
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: buildLocalResourceMap ....
14/11/05 13:39:3= 1 INFO yarn.YarnUtils: Registered file in LocalResources :: hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_141= 5189069846_0004/giraph-conf.xml
14/11/05 13:39:3= 1 INFO yarn.GiraphYarnClient: LIB JARS :
14/11/05 13:39:3= 1 INFO yarn.YarnUtils: Class path name .
14/11/05 13:39:3= 1 INFO yarn.YarnUtils: base path checking .
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name null
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking null
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/etc/hadoop
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/etc/hadoop=
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= common/lib/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/common/lib
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= common/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/common
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs/lib/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs/lib
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= hdfs/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/hdfs
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= yarn/lib/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/yarn/lib
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= yarn/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/yarn
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= mapreduce/lib/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/mapreduce/lib
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /opt/yarn/hadoop-2.5.1/share/hadoop/= mapreduce/*
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /opt/yarn/hadoop-2.5.1/share/hado= op/mapreduce
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: Class path name /contrib/capacity-scheduler/*.jar
14/11/05 14:42:5= 6 INFO yarn.YarnUtils: base path checking /contrib/capacity-scheduler
14/11/05 14:42:5= 6 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext for GiraphApplic= ationMaster launch container is populated.
14/11/05 14:42:5= 6 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/11/05 14:42:5= 7 INFO impl.YarnClientImpl: Submitted application application_1415189069846= _0004
14/11/05 14:42:5= 7 INFO yarn.GiraphYarnClient: Got new appId after submission :application_1= 415189069846_0004
14/11/05 14:42:5= 7 INFO yarn.GiraphYarnClient: GiraphApplicationMaster container request was= submitted to ResourceManager for job: Giraph: org.apache.giraph.examples.S= impleShortestPathsComputation
14/11/05 14:42:5= 7 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleShor= testPathsComputation, Elapsed: 0,82 secs
14/11/05 14:42:5= 7 INFO yarn.GiraphYarnClient: appattempt_1415189069846_0004_000001, State: = ACCEPTED, Containers used: 1
14/11/05 14:43:0= 1 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.examples.SimpleSho= rtestPathsComputation reports FAILED state, diagnostics show: Application a= pplication_1415189069846_0004 failed 2 times due to AM Container for appattempt_1415189069846_0004_000002 exite= d with  exitCode: 1 due to: Exception from container-launch: ExitCodeE= xception exitCode=3D1: 
ExitCodeExceptio= n exitCode=3D1: 
at org.apache.hadoop.util.Shell.runCommand(Shel= l.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:= 455)
at org.apache.hadoop.util.Shell$ShellCommandExe= cutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.De= faultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTa= sk.java:262)
at java.util.concurrent.ThreadPoolExecutor.runW= orker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Work= er.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
=
=
Container exited= with a non-zero exit code 1
.Failing this at= tempt.. Failing the application.
14/11/05 14:43:0= 1 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed cache directory = for Giraph job.
14/11/05 14:43:0= 1 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.= SimpleShortestPathsComputation: FAILED, total running time: 0 minutes, 3 se= conds.


Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <tripti@yahoo-inc.com> ha= scritto:

I faced the similar problem where the workflow just hangs. 
Can u see in ur logs if there is a task with id 0.
I made a local fix and I could give u the patch if u see a similar pro= blem and ur workflow is unable to find the zookeeper manager file and repea= ts the following message:
For task xyz, got file 'null'

Thanks,
Tripti
Sent from my iPhone

On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegro81@yahoo.it> wro= te:

Hi,
have you seen the hadoop logs?
Could you write here how are you running the command?

Thanks,
Alessandro

Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjing09@gmail.com> ha scrit= to:

Yeah, I add -yj option. The result is the same. Still erro= r here.

On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negr= o <alenegro81@yah= oo.it> wrote:
Hi Jing,
I=92m facing the same problem. Do you check the hadoop logs? In my cas= e I got: 
Could not find or load main class org.apache.giraph.yarn.GiraphApplica=
tionMaster
Moreover I don=92t know why my application hang for a long time here:<= /div>

14/11/05 17:11:1= 7 INFO yarn.YarnUtils: base path checking .

And then continue failing with your error into the output and the othe= r into the hadoop logs.

Did you test adding this?

-yj /Users/ale/e= xtprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-= with-dependencies.jar

Thanks,
Alessandro 


Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjing09@gmail.com> ha scrit= to:

Hi,

I install the Giraph with Yarn.
But when I run the program with following command:

 hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT= -for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar org.apache.giraph.Gira= phRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org= .apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/j/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueT= extOutputFormat -op /user/j/shortestpaths -w 1

The program exited with error:

14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: org.apache.girap= h.examples.SimpleShortestPathsComputation, Elapsed: 0.83 secs
14/11/05 10:25:14 INFO yarn.GiraphYarnClient: appattempt_1415203231170= _0002_000001, State: ACCEPTED, Containers used: 0
14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: org.apache.girap= h.examples.SimpleShortestPathsComputation, Elapsed: 4.84 secs
14/11/05 10:25:18 INFO yarn.GiraphYarnClient: appattempt_1415203231170= _0002_000001, State: ACCEPTED, Containers used: 0
14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: org.apache.girap= h.examples.SimpleShortestPathsComputation, Elapsed: 8.85 secs
14/11/05 10:25:22 INFO yarn.GiraphYarnClient: appattempt_1415203231170= _0002_000002, State: ACCEPTED, Containers used: 0
14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: org.apache.girap= h.examples.SimpleShortestPathsComputation, Elapsed: 12.86 secs
14/11/05 10:25:26 INFO yarn.GiraphYarnClient: appattempt_1415203231170= _0002_000002, State: ACCEPTED, Containers used: 0
14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: org.apache.gira= ph.examples.SimpleShortestPathsComputation reports FAILED state, diagnostic= s show: Application application_1415203231170_0002 failed 2 times due to AM= Container for appattempt_1415203231170_0002_000002 exited with  exitCode: 1 due to: Exception from container-launch.
Container id: container_1415203231170_0002_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=3D1: 
at org.apache.hadoop.util.= Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.= Shell.run(Shell.java:455)
at org.apache.hadoop.util.= Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.= server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContaine= rExecutor.java:196)
at org.apache.hadoop.yarn.= server.nodemanager.containermanager.launcher.ContainerLaunch.call(Container= Launch.java:299)
at org.apache.hadoop.yarn.= server.nodemanager.containermanager.launcher.ContainerLaunch.call(Container= Launch.java:81)
at java.util.concurrent.Fu= tureTask.run(FutureTask.java:262)
at java.util.concurrent.Th= readPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Th= read.java:745)


Anyone can help?
















--_000_D092AB9C1CB42triptiyahooinccom_--