Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E47A8DFD4 for ; Fri, 9 Nov 2012 13:06:47 +0000 (UTC) Received: (qmail 89338 invoked by uid 500); 9 Nov 2012 13:06:47 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 88970 invoked by uid 500); 9 Nov 2012 13:06:42 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 88931 invoked by uid 99); 9 Nov 2012 13:06:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 13:06:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of marcinbiczak@gmail.com designates 209.85.215.52 as permitted sender) Received: from [209.85.215.52] (HELO mail-la0-f52.google.com) (209.85.215.52) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Nov 2012 13:06:34 +0000 Received: by mail-la0-f52.google.com with SMTP id s15so2756331lag.11 for ; Fri, 09 Nov 2012 05:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=widkqdJpRuOhBaWwfQuRiItPO0FWsbAryr+/1vCHEXs=; b=xSop8BA/Vylz/G0Sra21RmdfPS1yLOpaEz5A1IEln3BEqxYTv0oYObYHMGP/dd/+wc 7q9cdTgv9auY8S7wt57FGNTCIW9e2thVGaTLml7+G1IqqtRozHzpARhD1iGKBOhKNVKN 6Flmg/RQMZwndgCrt+na4nIWFeBtCTcmXoA0lbpmKfRn+pYw9pIYGN4RtFqsOoks6dJX 16D3rw4L62tVQqJlsLTg2XtkEL2KTBj2n33ZN7OygFWrF/RZwEoR4H3nyoEYCqZzmTFU oWP7RuS0Y76vrlBoQ7xmU5N89Pl+mDu2CGI98M4q7B0gKiyBoxlR5jWWrjCHnwb0wxfH oz4A== Received: by 10.152.106.212 with SMTP id gw20mr10627417lab.8.1352466372161; Fri, 09 Nov 2012 05:06:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.114.0.142 with HTTP; Fri, 9 Nov 2012 05:05:51 -0800 (PST) In-Reply-To: References: From: Marcin Biczak Date: Fri, 9 Nov 2012 14:05:51 +0100 Message-ID: Subject: Re: java.lang.ClassNotFoundException for VertexOutputFormat To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=f46d0420a695aef25004ce0f9d13 X-Virus-Checked: Checked by ClamAV on apache.org --f46d0420a695aef25004ce0f9d13 Content-Type: text/plain; charset=ISO-8859-1 Hi again I have found a solution to my "problem". I simply added myJobs.jar to the HADOOP_CLASSPATH. Also "-libjars" parameter is not needed any more. But it still puzzles me why the previous "solution" "suddenly" stopped working (I haven't update the giraph revision etc). HADOOP_CLASSPATH contains girpah.jar and myJobs.jar regards marcin 2012/11/8 Marcin Biczak > Hi > > I have very strange problem regarding ClassNotFoundException for vertex > output format (my own implementation). When I run my program I get an > exception: > > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > org.test.giraph.utils.writers.generic.SamplingOutputFormat > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898) > at > org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134) > at > org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56) > at org.apache.hadoop.mapred.Task.initialize(Task.java:490) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352) > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:253) > > I have came across this exception earlier and I've solved it with the use > of "-libjars" parameter in which I pass the path to > giraph-0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I > execute). This solution has worked very nice for me, I used it in a number > of programs (with my own InputFormat and OutputFormat), till today. I've > created a small program (TestVertex) to test one solution and I used a > NewOutputFormat created for the test program. When I received > ClassNotFoundException I thought I have made a mistake somewhere, so I've > checked. I couldn't find any bugs, so I replaced the NewOutputFormat with a > one used across other programs (OldOutputFormat). I got the same exception. > for the old one as well. I have cleaned and build the whole myJob.jar and I > was still getting the exception (both for New and Old). So I tested the > original program (same myJob.jar) in which I'm using the OldOutputFormat > and the program works. Since then I tried couple of OutputFormat and none > of them worked (ClassNotFoundException). Then I commented the whole "body" > of TestVertex and I've paste some old working code (with OldOutputFormat). > The only difference between TestVertex and old working code is the class > name. Final I received the ClassNotFoundException again (again the original > code works, I've tested it). It looks like only the TestVertex class has > this problem, no matter what OutputFormat I will use I am always getting > the exception (all other programs which use my OutputFormats work > correctly). > > Another interesting aspect is the fact that if I use Giraph provided > OutputFormat (for example JsonBase64VertexOutputFormat) I get a > ClassNotFoundException but for the TestVertex. I've checked the myJob.jar > and all mentioned classes are present in the jar in correct places. > > I run out of ideas any suggestions will be greatly appreciated. > > Some conf info > I am using an older Giraph revision (1340869) and Hadoop 0.20.203 in > pseudo-distributed mode. > I have created the HADOOP_CLASSPATH which holds path to > giraph-0.2-SNAPSHOT-jar-with-dependencies.jar. > I execute job in the following way: hadoop jar myJob.jar > org.test.giraph.Test -libjars > /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/myJob.jar > /path/to/input /path/to/output > > regards > marcin > -- martin --f46d0420a695aef25004ce0f9d13 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi again

I have found a solution to my "problem". I simply= added myJobs.jar to the HADOOP_CLASSPATH. Also "-libjars" parame= ter is not needed any more. But it still puzzles me why the previous "= solution" "suddenly" stopped working (I haven't update t= he giraph revision etc).

HADOOP_CLASSPATH contains girpah.jar and myJobs.jar

regards
m= arcin

2012/11/8 Marcin Biczak <ma= rcinbiczak@gmail.com>
Hi

I have very strange problem regard= ing ClassNotFoundException for vertex output format (my own implementation)= . When I run my program I get an exception:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.Class= NotFoundException: org.test.giraph.utils.writers.generic.SamplingOutputForm= at
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.conf.Configuration.getClass(Conf= iguration.java:898)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.giraph.graph.Bsp= Utils.getVertexOutputFormatClass(BspUtils.java:134)
=A0=A0=A0=A0=A0=A0= =A0 at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFo= rmat.java:56)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.Task.initialize(Task.java= :490)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.MapTask.run(MapT= ask.java:352)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.mapred.Child$4.= run(Child.java:259)
=A0=A0=A0=A0=A0=A0=A0 at java.security.AccessControl= ler.doPrivileged(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at javax.security.auth.Subject.doAs(Subject.java:396)=
=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.security.UserGroupInformatio= n.doAs(UserGroupInformation.java:1059)
=A0=A0=A0=A0=A0=A0=A0 at org.apac= he.hadoop.mapred.Child.main(Child.java:253)

I have came across this exception earlier and I've solved it with t= he use of "-libjars" parameter in which I pass the path to giraph= -0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I execute). Th= is solution has worked very nice for me, I used it in a number of programs = (with my own InputFormat and OutputFormat), till today. I've created a = small program (TestVertex) to test one solution and I used a NewOutputForma= t created for the test program. When I received ClassNotFoundException I th= ought I have made a mistake somewhere, so I've checked. I couldn't = find any bugs, so I replaced the NewOutputFormat with a one used across oth= er programs (OldOutputFormat). I got the same exception. for the old one as= well. I have cleaned and build the whole myJob.jar and I was still getting= the exception (both for New and Old). So I tested the original program (sa= me myJob.jar) in which I'm using the OldOutputFormat and the program wo= rks. Since then I tried couple of OutputFormat and none of them worked (Cla= ssNotFoundException). Then I commented the whole "body" of TestVe= rtex and I've paste some old working code (with OldOutputFormat). The o= nly difference between TestVertex and old working code is the class name. F= inal I received the ClassNotFoundException again (again the original code w= orks, I've tested it). It looks like only the TestVertex class has this= problem, no matter what OutputFormat I will use I am always getting the ex= ception (all other programs which use my OutputFormats work correctly).

Another interesting aspect is the fact that if I use Giraph provided Ou= tputFormat (for example JsonBase64VertexOutputFormat) I get a ClassNotFound= Exception but for the TestVertex. I've checked the myJob.jar and all me= ntioned classes are present in the jar in correct places.

I run out of ideas any suggestions will be greatly appreciated.

= Some conf info
I am using an older Giraph revision (1340869) and Hadoop = 0.20.203 in pseudo-distributed mode.
I have created the HADOOP_CLASSPAT= H which holds path to giraph-0.2-SNAPSHOT-jar-with-dependencies.jar.
I execute job in the following way: hadoop jar myJob.jar org.test.giraph.Te= st -libjars /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to= /myJob.jar /path/to/input /path/to/output <job params>

regards=
marcin



--
martin --f46d0420a695aef25004ce0f9d13--