giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Sembium <vivek.semb...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Mon, 26 Aug 2013 09:23:55 GMT
Yes for the zookeeper problem I passed a seperate jar through -libjars
command. If I use additional jars zookeeper fails.
On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.martella@gmail.com>
wrote:

> there must be a misunderstanding. i was referring to the zookeeper problem.
>
>
> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>
>> No. I added my files(it was just a copy of one of the example program to
>> a different package) to the jar files of giraph. But it was still giving me
>> classNotFoundException. Can you give me some simple example program with
>> instructions on how to deploy it. So I can start playing with giraph and
>> make changes to the program and learn, then start working on my project in
>> giraph.
>> I will be very thankful if you can help me with this.
>>
>> Thanking you
>> -Vivek Sembium
>>
>>
>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
>> claudio.martella@gmail.com> wrote:
>>
>>> but you were still using an additional jar added through -libjars, right?
>>>
>>>
>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>>>
>>>> @Claudio Martella Your solution didnt work either. I basically tried
>>>> copying the pageRankBenchmark to my own package, renamed the package. It
>>>> compiles fine with giraph. But I couldnt run it even if I add those files
>>>> to giraph jar before deployment. Help?
>>>>
>>>>
>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella <
>>>> claudio.martella@gmail.com> wrote:
>>>>
>>>>> you have this problem when you use two jars (one with giraph and one
>>>>> with your classes) instead of a single fat-jar, correct? I tracked the
same
>>>>> problem a few weeks ago, basically zookeeper is run passing the wrong
jar.
>>>>>
>>>>>
>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <
>>>>> vivek.sembium@gmail.com> wrote:
>>>>>
>>>>>> Thank you for your suggestion. It worked. Its not giving class not
>>>>>> found exception. But its giving me a new error
>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found that
>>>>>> its unable to connect to zookeeper service.
>>>>>>
>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries!
>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers:
>>>>>> Failed to connect in 10 tries!
>>>>>>  at
>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>>>> at
>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>>>  at
>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>>>> ... 7 more
>>>>>>
>>>>>> Immediately I ran page rank benchmark and it executed successfully
>>>>>> both from giraph in lib directory and also from giraphs own directory.
>>>>>>
>>>>>> Can you give me a very simple java program(finding maximum in a graph
>>>>>> or simple page rank program) in giraph along with its jar file and
input
>>>>>> files  which I can place in my lib directory of hadoop and test if
its
>>>>>> working. And also the command to execute it. This should be added
in the
>>>>>> documentation as new comers can quickly setup giraph and concentrate
on
>>>>>> their project.
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ <
>>>>>> emre.aladag@agmlab.com> wrote:
>>>>>>
>>>>>>> It isn't asking for edge input. It says make sure you don't need
it.
>>>>>>> A warning for the case you may have forgotten to give edge input
when you
>>>>>>> really needed.
>>>>>>>
>>>>>>> The cause of your error is what I'm wondering nowadays. I'm having
a
>>>>>>> similar problem. Currently I'm using a workaround: put all the
jars
>>>>>>> (giraph-core and my module giraph-nutch) in the lib folder of
hadoop. Then
>>>>>>> it works. But there should be a clean way of doing this.
>>>>>>>
>>>>>>> I should be able to say hadoop jar fat.jar ...
>>>>>>>
>>>>>>> Any help appreciated.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>>>> *Kime: *user@giraph.apache.org
>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49
>>>>>>> *Konu: *Re: Help needed for Running my own java programs in Giraph
>>>>>>>
>>>>>>>
>>>>>>> I tried with and without exporting hadoop classpath. I get the
same
>>>>>>> error.
>>>>>>>
>>>>>>> Here's the command that I tried
>>>>>>> hadoop jar
>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute
-vif
>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>>>
>>>>>>>
>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input
>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>>>         at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>         at
>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>>>         at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>>>         at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>         at
>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>>>         at
>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>>>         at
>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>>>         at
>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>>>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>> Method)
>>>>>>>         at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>         at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>         at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>         at
>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>         ... 26 more
>>>>>>>
>>>>>>>
>>>>>>> I gave the graph input as JSON file. Why does it ask for edge
input
>>>>>>> file again?
>>>>>>>
>>>>>>>
>>>>>>> Why isnt this working?
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando <
>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Vivek,
>>>>>>>>
>>>>>>>> Sorry for the hasty response, I assumed the quick start guide
>>>>>>>> covered that (I guess it doesn't). What you need to do first
is put all of
>>>>>>>> your compiled code into a jar file.  Then, you'll be utilizing
the hadoop
>>>>>>>> option -libjars.  You also need to make sure that you set
the hadoop
>>>>>>>> classpath to include the giraph jar you are using as well
as your own jar.
>>>>>>>> I actually asked a similar question not too long ago; prior
to this, I was
>>>>>>>> just combining everything into one jar.
>>>>>>>>
>>>>>>>>
>>>>>>>> Your code should pretty much follow this format, which is
copied
>>>>>>>> and paster from a previous response by Claudio:
>>>>>>>>
>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D
option
>>>>>>>> ]* -libjars your.jar your.package.ComputationClass [ GiraphRunner
>>>>>>>> option e.g. -vip -vif etc. ]*
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium <
>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thank You for your response. I could run all the example
programs.
>>>>>>>>> I want to place my development code in a different directory
other than the
>>>>>>>>> giraph directory. I copied the SimplePageRankComputation.java
to a
>>>>>>>>> different directory, compiled it(with a different package
name). But I
>>>>>>>>> couldnt replace the "org.apache.giraph.examples.SimpleShortestPathsComputation"
>>>>>>>>> with my class file. It gives me a ClassNotFoundException.
Please Guide me
>>>>>>>>> so that I can run any class file with all required methods
(GIven the
>>>>>>>>> location of that file).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando <
>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> A "Quick Start" guide was recently added to the Giraph
website.
>>>>>>>>>> Now learning Giraph is easier than ever!  Here:
>>>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium <
>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am a new user in giraph. I could run all example
programs in
>>>>>>>>>>> Giraph. Can someone please guide me on how to
run my own java program in
>>>>>>>>>>> giraph (Assuming I have my input file in jason
format in HDFS).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Kyle Orlando
>>>>>>>>>> Computer Engineering Major
>>>>>>>>>> University of Maryland
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kyle Orlando
>>>>>>>> Computer Engineering Major
>>>>>>>> University of Maryland
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>    Claudio Martella
>>>>>    claudio.martella@gmail.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>    Claudio Martella
>>>    claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>

Mime
View raw message