Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C718B10923 for ; Mon, 15 Jul 2013 18:03:01 +0000 (UTC) Received: (qmail 50529 invoked by uid 500); 15 Jul 2013 18:03:00 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 50390 invoked by uid 500); 15 Jul 2013 18:03:00 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 50372 invoked by uid 99); 15 Jul 2013 18:02:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 18:02:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of apache.mailbox@gmail.com designates 209.85.128.171 as permitted sender) Received: from [209.85.128.171] (HELO mail-ve0-f171.google.com) (209.85.128.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Jul 2013 18:02:52 +0000 Received: by mail-ve0-f171.google.com with SMTP id b10so10146272vea.2 for ; Mon, 15 Jul 2013 11:02:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=ul5WIK+ut/IJl/LWGYgyLp06PDv1In0IlUgp+CqJoUk=; b=UXatNxZhOr/bn/SxDNJ8dIE/zszxC4odbiYTv58MsRlYpgiWUsLT/xPftSfxFsUqXQ +2CYiG0MWJmNEH3yIJwEQEayd3GMjVT8ZJfiMvfzgdfWWoqCaHP3I26XMpIJWb0jgdXM rE3KWJKerGZCnhweR9Ny/kr7hk0X3cn426CorFZDYzqOR5IFqEhTkNzkdYmVr+JhI1tN Tz/Cn1qaBnQwz/k+nZkSCLHOus/lsy3qsqWgaL7Hyx5v9lvibsTfBGIU/8FOxXhw1MCN IZG3eOfDdkh44OdRjIdEG7c3CGC8qEnWlH7MzhZ73OciEOwEmD6ZGlZOYUXnLGLKbLnR GU3w== MIME-Version: 1.0 X-Received: by 10.220.181.69 with SMTP id bx5mr11585100vcb.71.1373911351289; Mon, 15 Jul 2013 11:02:31 -0700 (PDT) Received: by 10.52.64.231 with HTTP; Mon, 15 Jul 2013 11:02:31 -0700 (PDT) In-Reply-To: <51D97B39.9030709@googlemail.com> References: <51D97B39.9030709@googlemail.com> Date: Mon, 15 Jul 2013 11:02:31 -0700 Message-ID: Subject: Re: GiraphApplicationMaster not found (and other newbie questions) From: Eli Reisman To: user@giraph.apache.org, nicholaskar+giraph@googlemail.com Content-Type: multipart/alternative; boundary=001a1132f30c0beab904e190aa5b X-Virus-Checked: Checked by ClamAV on apache.org --001a1132f30c0beab904e190aa5b Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable hi nicholas, Looks like your build was not pulling in the Giraph yarn/ package classes, probably you build using something like: mvn -Phadoop_2.0.5 clean install try instead: mvn -Dhadoop.version=3D2.0.5-alpha -Phadoop_yarn clean install (assuming you have applied the patch for 2.0.5-alpha builds on Giraph I htink its GIRAPH-688, otherwise, the hadoop_yarn profile only builds against 2.0.3-alpha hadoop) On Sun, Jul 7, 2013 at 7:29 AM, Nicholas Karkoulias wrote: > Hello everyone. > > This is my first message to an Apache mailing list, so please excuse (and > correct) possibly incorrect usage or netiquette issues :) > > I am a newbie to both Hadoop and Giraph (but not to Linux). After > overcoming _several_ configuration-related hurdles, I have successfully > built both (versions Hadoop 2.0.5-alpha and Giraph 1.1.0-SNAPSHOT, using > the patch for issue #688), and seem to have a properly working HDFS/Hadoo= p > installation in pseudo-distributed mode. I can run Hadoop code just fine. > > However, Giraph fails during the computation (seemingly just before > returning or writing the result =96 the time elapsed before the crash > differs, depending on which example I try to run). See below for the erro= r. > I don't know whether it is a bug, or me doing something wrong. > > I'm using the YARN-enabled version of Giraph (and, thus, an external > ZooKeeper service), assuming that Giraph will completely move to YARN > _eventually_. (Is that correct?) > > Also, two other (somewhat unrelated) questions: > (1) When is the file conf/giraph-site.xml actually parsed/used? Is it rea= d > by Hadoop? Should it be copied somewhere? I tried setting the ZooKeeper > host:port in that file (giraph.zkList), but it was ignored and I finally > had to add the property in the command line shown below... Any relevant > documentation? (In general, documentation for certain Hadoop features, su= ch > as the configuration files, seems to be lacking...) > (2) What's the correct process to submit a patch with a really simple typ= o > correction in a string? (perhaps I should just contact the file author = =96 > it's nothing important) > > I'll append the shell commands I used (after $) and the output at the end > of this message. > > Thank you in advance, > Nicholas > > > $ function giraphrunner(){ hadoop jar /tmp/software/giraph/giraph-** > examples/target/giraph-**examples-1.1.0-SNAPSHOT-for-** > hadoop-2.0.5-alpha-jar-with-**dependencies.jar > org.apache.giraph.GiraphRunner -Dgiraph.zkList=3Dlocalhost:2181 "$@"; } > > $ time giraphrunner org.apache.giraph.examples.**SimplePageRankComputatio= n > -vif org.apache.giraph.io.formats.**JsonLongDoubleFloatDoubleVerte**xInpu= tFormat > -vip /dir/tiny_graph.txt -of org.apache.giraph.io.formats.**IdWithValueTe= xtOutputFormat > -op /dir/simplepagerank -w 1 > 13/07/07 15:06:15 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://localhost:9000/dir/**simplepagerank > 13/07/07 15:06:15 INFO service.AbstractService: Service:org.apache.hadoop= . > **yarn.client.YarnClientImpl is inited. > 13/07/07 15:06:15 INFO service.AbstractService: Service:org.apache.hadoop= . > **yarn.client.YarnClientImpl is started. > 13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Defaulting per-task heap > size to 1024MB. > 13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Obtained new Application ID= : > application_1372875746593_0018 > 13/07/07 15:06:15 WARN conf.Configuration: mapred.job.id is deprecated. > Instead, use mapreduce.job.id > 13/07/07 15:06:15 WARN conf.Configuration: mapred.output.dir is > deprecated. Instead, use mapreduce.output.**fileoutputformat.outputdir > 13/07/07 15:06:15 INFO yarn.YarnUtils: Registered file in LocalResources: > giraph-conf.xml > 13/07/07 15:06:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContex= t > for GiraphApplicationMaster launch container is populated. > 13/07/07 15:06:16 INFO client.YarnClientImpl: Submitted application > application_1372875746593_0018 to ResourceManager at localhost/ > 127.0.0.1:8032 > 13/07/07 15:06:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.**SimplePageRankComputation > 13/07/07 15:06:17 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.**SimplePageRankComputation, Elapsed: 0.85 sec= s > 13/07/07 15:06:17 INFO yarn.GiraphYarnClient: > appattempt_1372875746593_0018_**000001, State: ACCEPTED, Containers used: > 1 > 13/07/07 15:06:18 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.**SimplePageRankComputation reports FAILED > state, diagnostics show: Application application_1372875746593_0018 faile= d > 1 times due to AM Container for appattempt_1372875746593_0018_**000001 > exited with exitCode: 1 due to: > .Failing this attempt.. Failing the application. > 13/07/07 15:06:18 INFO yarn.GiraphYarnClient: Cleaning up HDFS distribute= d > cache directory for Giraph job. > 13/07/07 15:06:18 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.**SimplePageRankComputation: FAILED, total > running time: 0 minutes, 1 seconds. > > real 0m8.392s > user 0m8.825s > sys 0m1.492s > > $ cat software/hadoop-2.0.5-alpha/**logs/userlogs/application_** > 1372875746593_0018/container_**1372875746593_0018_01_000001/** > gam-stderr.log > Exception in thread "main" java.lang.**NoClassDefFoundError: > org/apache/giraph/yarn/**GiraphApplicationMaster > Caused by: java.lang.**ClassNotFoundException: org.apache.giraph.yarn.** > GiraphApplicationMaster > at java.net.URLClassLoader$1.run(**URLClassLoader.java:217) > at java.security.**AccessController.doPrivileged(**Native Method) > at java.net.URLClassLoader.**findClass(URLClassLoader.java:**205) > at java.lang.ClassLoader.**loadClass(ClassLoader.java:**321) > at sun.misc.Launcher$**AppClassLoader.loadClass(** > Launcher.java:294) > at java.lang.ClassLoader.**loadClass(ClassLoader.java:**266) > Could not find the main class: org.apache.giraph.yarn.**GiraphApplication= Master. > Program will exit. > > --001a1132f30c0beab904e190aa5b Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
hi nicholas,

Looks like your build was = not pulling in the Giraph yarn/ package classes, probably you build using s= omething like:

mvn -Phadoop_2.0.5 clean inst= all

try instead:

mvn -Dhadoop.version=3D2.0.5-alpha -Phadoop_yarn clean install
=

(assuming you have applied the patch for 2.= 0.5-alpha builds on Giraph I htink its GIRAPH-688,=A0otherwise, the hadoop_= yarn profile only builds against 2.0.3-alpha hadoop)


=

On Sun, Jul 7, 2013 at 7:29 AM, Nicholas= Karkoulias <nicholaskar@gmail.com> wrote:
Hello everyone.

This is my first message to an Apache mailing list, so please excuse (and c= orrect) possibly incorrect usage or netiquette issues :)

I am a newbie to both Hadoop and Giraph (but not to Linux). After overcomin= g _several_ configuration-related hurdles, I have successfully built both (= versions Hadoop 2.0.5-alpha and Giraph 1.1.0-SNAPSHOT, using the patch for = issue #688), and seem to have a properly working HDFS/Hadoop installation i= n pseudo-distributed mode. I can run Hadoop code just fine.

However, Giraph fails during the computation (seemingly just before returni= ng or writing the result =96 the time elapsed before the crash differs, dep= ending on which example I try to run). See below for the error. I don't= know whether it is a bug, or me doing something wrong.

I'm using the YARN-enabled version of Giraph (and, thus, an external Zo= oKeeper service), assuming that Giraph will completely move to YARN _eventu= ally_. (Is that correct?)

Also, two other (somewhat unrelated) questions:
(1) When is the file conf/giraph-site.xml actually parsed/used? Is it read = by Hadoop? Should it be copied somewhere? I tried setting the ZooKeeper hos= t:port in that file (giraph.zkList), but it was ignored and I finally had t= o add the property in the command line shown below... Any relevant document= ation? (In general, documentation for certain Hadoop features, such as the = configuration files, seems to be lacking...)
(2) What's the correct process to submit a patch with a really simple t= ypo correction in a string? (perhaps I should just contact the file author = =96 it's nothing important)

I'll append the shell commands I used (after $) and the output at the e= nd of this message.

Thank you in advance,
Nicholas


$ function giraphrunner(){ hadoop jar /tmp/software/giraph/giraph-ex= amples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.0.5= -alpha-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dgi= raph.zkList=3Dlocalhost:2181 "$@"; }

$ time giraphrunner org.apache.giraph.examples.SimplePageRankComputa= tion -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVert= exInputFormat -vip /dir/tiny_graph.txt -of org.apache.giraph.io.form= ats.IdWithValueTextOutputFormat -op /dir/simplepagerank -w 1
13/07/07 15:06:15 INFO utils.ConfigurationUtils: No edge input format speci= fied. Ensure your InputFormat does not require one.
13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Final output path is: hdfs://= localhost:9000/dir/simplepagerank
13/07/07 15:06:15 INFO service.AbstractService: Service:org.apache.hadoop.<= u>yarn.client.YarnClientImpl is inited.
13/07/07 15:06:15 INFO service.AbstractService: Service:org.apache.hadoop.<= u>yarn.client.YarnClientImpl is started.
13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Defaulting per-task heap size= to 1024MB.
13/07/07 15:06:15 INFO yarn.GiraphYarnClient: Obtained new Application ID: = application_1372875746593_0018
13/07/07 15:06:15 WARN conf.Configuration: mapred.job.id is deprecated. Instead, use mapreduce.job.id
13/07/07 15:06:15 WARN conf.Configuration: mapred.output.dir is deprecated.= Instead, use mapreduce.output.fileoutputformat.outputdir
13/07/07 15:06:15 INFO yarn.YarnUtils: Registered file in LocalResources: g= iraph-conf.xml
13/07/07 15:06:16 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext = for GiraphApplicationMaster launch container is populated.
13/07/07 15:06:16 INFO client.YarnClientImpl: Submitted application applica= tion_1372875746593_0018 to ResourceManager at localhost/127.0.0.1:8032
13/07/07 15:06:16 INFO yarn.GiraphYarnClient: GiraphApplicationMaster conta= iner request was submitted to ResourceManager for job: Giraph: org.apache.g= iraph.examples.SimplePageRankComputation
13/07/07 15:06:17 INFO yarn.GiraphYarnClient: Giraph: org.apache.giraph.exa= mples.SimplePageRankComputation, Elapsed: 0.85 secs
13/07/07 15:06:17 INFO yarn.GiraphYarnClient: appattempt_1372875746593_0018= _000001, State: ACCEPTED, Containers used: 1
13/07/07 15:06:18 ERROR yarn.GiraphYarnClient: Giraph: org.apache.giraph.ex= amples.SimplePageRankComputation reports FAILED state, diagnostics s= how: Application application_1372875746593_0018 failed 1 times due to AM Co= ntainer for appattempt_1372875746593_0018_000001 exited with =A0exit= Code: 1 due to:
.Failing this attempt.. Failing the application.
13/07/07 15:06:18 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed = cache directory for Giraph job.
13/07/07 15:06:18 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.= giraph.examples.SimplePageRankComputation: FAILED, total running tim= e: 0 minutes, 1 seconds.

real =A0 =A00m8.392s
user =A0 =A00m8.825s
sys =A0 =A0 0m1.492s

$ cat software/hadoop-2.0.5-alpha/logs/userlogs/application_1= 372875746593_0018/container_1372875746593_0018_01_000001/gam-= stderr.log
Exception in thread "main" java.lang.NoClassDefFoundError:= org/apache/giraph/yarn/GiraphApplicationMaster
Caused by: java.lang.ClassNotFoundException: org.apache.giraph.yarn.= GiraphApplicationMaster
=A0 =A0 =A0 =A0 at java.net.URLClassLoader$1.run(URLClassLoader.java= :217)
=A0 =A0 =A0 =A0 at java.security.AccessController.doPrivileged(Native Method)
=A0 =A0 =A0 =A0 at java.net.URLClassLoader.findClass(URLClassLoader.= java:205)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:= 321)
=A0 =A0 =A0 =A0 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
=A0 =A0 =A0 =A0 at java.lang.ClassLoader.loadClass(ClassLoader.java:= 266)
Could not find the main class: org.apache.giraph.yarn.GiraphApplicat= ionMaster. Program will exit.


--001a1132f30c0beab904e190aa5b--