giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Sembium <vivek.semb...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Mon, 26 Aug 2013 09:14:40 GMT
No. I added my files(it was just a copy of one of the example program to a
different package) to the jar files of giraph. But it was still giving me
classNotFoundException. Can you give me some simple example program with
instructions on how to deploy it. So I can start playing with giraph and
make changes to the program and learn, then start working on my project in
giraph.
I will be very thankful if you can help me with this.

Thanking you
-Vivek Sembium


On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
claudio.martella@gmail.com> wrote:

> but you were still using an additional jar added through -libjars, right?
>
>
> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>
>> @Claudio Martella Your solution didnt work either. I basically tried
>> copying the pageRankBenchmark to my own package, renamed the package. It
>> compiles fine with giraph. But I couldnt run it even if I add those files
>> to giraph jar before deployment. Help?
>>
>>
>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella <
>> claudio.martella@gmail.com> wrote:
>>
>>> you have this problem when you use two jars (one with giraph and one
>>> with your classes) instead of a single fat-jar, correct? I tracked the same
>>> problem a few weeks ago, basically zookeeper is run passing the wrong jar.
>>>
>>>
>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>>>
>>>> Thank you for your suggestion. It worked. Its not giving class not
>>>> found exception. But its giving me a new error
>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found that its
>>>> unable to connect to zookeeper service.
>>>>
>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>>>> onlineZooKeeperServers: Failed to connect in 10 tries!
>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers:
>>>> Failed to connect in 10 tries!
>>>>  at
>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>> at
>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>  at
>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>> ... 7 more
>>>>
>>>> Immediately I ran page rank benchmark and it executed successfully both
>>>> from giraph in lib directory and also from giraphs own directory.
>>>>
>>>> Can you give me a very simple java program(finding maximum in a graph
>>>> or simple page rank program) in giraph along with its jar file and input
>>>> files  which I can place in my lib directory of hadoop and test if its
>>>> working. And also the command to execute it. This should be added in the
>>>> documentation as new comers can quickly setup giraph and concentrate on
>>>> their project.
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ <
>>>> emre.aladag@agmlab.com> wrote:
>>>>
>>>>> It isn't asking for edge input. It says make sure you don't need it.
A
>>>>> warning for the case you may have forgotten to give edge input when you
>>>>> really needed.
>>>>>
>>>>> The cause of your error is what I'm wondering nowadays. I'm having a
>>>>> similar problem. Currently I'm using a workaround: put all the jars
>>>>> (giraph-core and my module giraph-nutch) in the lib folder of hadoop.
Then
>>>>> it works. But there should be a clean way of doing this.
>>>>>
>>>>> I should be able to say hadoop jar fat.jar ...
>>>>>
>>>>> Any help appreciated.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>> *Kime: *user@giraph.apache.org
>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49
>>>>> *Konu: *Re: Help needed for Running my own java programs in Giraph
>>>>>
>>>>>
>>>>> I tried with and without exporting hadoop classpath. I get the same
>>>>> error.
>>>>>
>>>>> Here's the command that I tried
>>>>> hadoop jar
>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute -vif
>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>
>>>>>
>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input format
>>>>> specified. Ensure your InputFormat does not require one.
>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format
>>>>> specified. Ensure your OutputFormat does not require one.
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>         at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>         at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>         at
>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>         at
>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>         at
>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>         at
>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>         at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>         at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>         at
>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>         ... 26 more
>>>>>
>>>>>
>>>>> I gave the graph input as JSON file. Why does it ask for edge input
>>>>> file again?
>>>>>
>>>>>
>>>>> Why isnt this working?
>>>>>
>>>>>
>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando <
>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>
>>>>>> Hi Vivek,
>>>>>>
>>>>>> Sorry for the hasty response, I assumed the quick start guide covered
>>>>>> that (I guess it doesn't). What you need to do first is put all of
your
>>>>>> compiled code into a jar file.  Then, you'll be utilizing the hadoop
option
>>>>>> -libjars.  You also need to make sure that you set the hadoop classpath
to
>>>>>> include the giraph jar you are using as well as your own jar. I actually
>>>>>> asked a similar question not too long ago; prior to this, I was just
>>>>>> combining everything into one jar.
>>>>>>
>>>>>>
>>>>>> Your code should pretty much follow this format, which is copied
and
>>>>>> paster from a previous response by Claudio:
>>>>>>
>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D option
]*
>>>>>> -libjars your.jar your.package.ComputationClass [ GiraphRunner
>>>>>> option e.g. -vip -vif etc. ]*
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium <
>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>
>>>>>>> Thank You for your response. I could run all the example programs.
I
>>>>>>> want to place my development code in a different directory other
than the
>>>>>>> giraph directory. I copied the SimplePageRankComputation.java
to a
>>>>>>> different directory, compiled it(with a different package name).
But I
>>>>>>> couldnt replace the "org.apache.giraph.examples.SimpleShortestPathsComputation"
>>>>>>> with my class file. It gives me a ClassNotFoundException. Please
Guide me
>>>>>>> so that I can run any class file with all required methods (GIven
the
>>>>>>> location of that file).
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando <
>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>
>>>>>>>> A "Quick Start" guide was recently added to the Giraph website.
Now
>>>>>>>> learning Giraph is easier than ever!  Here:
>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium <
>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I am a new user in giraph. I could run all example programs
in
>>>>>>>>> Giraph. Can someone please guide me on how to run my
own java program in
>>>>>>>>> giraph (Assuming I have my input file in jason format
in HDFS).
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kyle Orlando
>>>>>>>> Computer Engineering Major
>>>>>>>> University of Maryland
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kyle Orlando
>>>>>> Computer Engineering Major
>>>>>> University of Maryland
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>    Claudio Martella
>>>    claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>

Mime
View raw message