giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Mon, 26 Aug 2013 15:00:12 GMT
you mean by running zookeeper independently?


On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:

> We were also experiencing similar problems when specifying -libjars as
> opposed to just using a fat jar.  I believe we fixed it by setting the
> giraph.zkList property, but this only appears to work when we list one node
> as a zookeeper.
>
>
> On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella <
> claudio.martella@gmail.com> wrote:
>
>> OK, then I'm going to open an issue for that.
>>
>>
>> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>>
>>> Yes for the zookeeper problem I passed a seperate jar through -libjars
>>> command. If I use additional jars zookeeper fails.
>>>  On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.martella@gmail.com>
>>> wrote:
>>>
>>>> there must be a misunderstanding. i was referring to the zookeeper
>>>> problem.
>>>>
>>>>
>>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <
>>>> vivek.sembium@gmail.com> wrote:
>>>>
>>>>> No. I added my files(it was just a copy of one of the example program
>>>>> to a different package) to the jar files of giraph. But it was still
giving
>>>>> me classNotFoundException. Can you give me some simple example program
with
>>>>> instructions on how to deploy it. So I can start playing with giraph
and
>>>>> make changes to the program and learn, then start working on my project
in
>>>>> giraph.
>>>>> I will be very thankful if you can help me with this.
>>>>>
>>>>> Thanking you
>>>>> -Vivek Sembium
>>>>>
>>>>>
>>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
>>>>> claudio.martella@gmail.com> wrote:
>>>>>
>>>>>> but you were still using an additional jar added through -libjars,
>>>>>> right?
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <
>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>
>>>>>>> @Claudio Martella Your solution didnt work either. I basically
tried
>>>>>>> copying the pageRankBenchmark to my own package, renamed the
package. It
>>>>>>> compiles fine with giraph. But I couldnt run it even if I add
those files
>>>>>>> to giraph jar before deployment. Help?
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella <
>>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>>
>>>>>>>> you have this problem when you use two jars (one with giraph
and
>>>>>>>> one with your classes) instead of a single fat-jar, correct?
I tracked the
>>>>>>>> same problem a few weeks ago, basically zookeeper is run
passing the wrong
>>>>>>>> jar.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <
>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thank you for your suggestion. It worked. Its not giving
class not
>>>>>>>>> found exception. But its giving me a new error
>>>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection
I found that
>>>>>>>>> its unable to connect to zookeeper service.
>>>>>>>>>
>>>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>>>>>> exception onlineZooKeeperServers: Failed to connect in
10 tries!
>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>>>>>>  at
>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>>>>> at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>>>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>>>>> Caused by: java.lang.IllegalStateException:
>>>>>>>>> onlineZooKeeperServers: Failed to connect in 10 tries!
>>>>>>>>>  at
>>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>>>>>>  at
>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>>>>>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>>>>>>> ... 7 more
>>>>>>>>>
>>>>>>>>> Immediately I ran page rank benchmark and it executed
successfully
>>>>>>>>> both from giraph in lib directory and also from giraphs
own directory.
>>>>>>>>>
>>>>>>>>> Can you give me a very simple java program(finding maximum
in a
>>>>>>>>> graph or simple page rank program) in giraph along with
its jar file and
>>>>>>>>> input files  which I can place in my lib directory of
hadoop and test if
>>>>>>>>> its working. And also the command to execute it. This
should be added in
>>>>>>>>> the documentation as new comers can quickly setup giraph
and concentrate on
>>>>>>>>> their project.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ <
>>>>>>>>> emre.aladag@agmlab.com> wrote:
>>>>>>>>>
>>>>>>>>>> It isn't asking for edge input. It says make sure
you don't need
>>>>>>>>>> it. A warning for the case you may have forgotten
to give edge input when
>>>>>>>>>> you really needed.
>>>>>>>>>>
>>>>>>>>>> The cause of your error is what I'm wondering nowadays.
I'm
>>>>>>>>>> having a similar problem. Currently I'm using a workaround:
put all the
>>>>>>>>>> jars (giraph-core and my module giraph-nutch) in
the lib folder of hadoop.
>>>>>>>>>> Then it works. But there should be a clean way of
doing this.
>>>>>>>>>>
>>>>>>>>>> I should be able to say hadoop jar fat.jar ...
>>>>>>>>>>
>>>>>>>>>> Any help appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------------------
>>>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>>>>>>> *Kime: *user@giraph.apache.org
>>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49
>>>>>>>>>> *Konu: *Re: Help needed for Running my own java programs
in
>>>>>>>>>> Giraph
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I tried with and without exporting hadoop classpath.
I get the
>>>>>>>>>> same error.
>>>>>>>>>>
>>>>>>>>>> Here's the command that I tried
>>>>>>>>>> hadoop jar
>>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>>>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute
-vif
>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No edge input
>>>>>>>>>> format specified. Ensure your InputFormat does not
require one.
>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No output format
>>>>>>>>>> specified. Ensure your OutputFormat does not require
one.
>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>>>>>>         at java.lang.ClassLoader.defineClass1(Native
Method)
>>>>>>>>>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>>>>>>         at
>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>         at
>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>         at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>>         at
>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>         at
>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>         ... 26 more
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I gave the graph input as JSON file. Why does it
ask for edge
>>>>>>>>>> input file again?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Why isnt this working?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando <
>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Vivek,
>>>>>>>>>>>
>>>>>>>>>>> Sorry for the hasty response, I assumed the quick
start guide
>>>>>>>>>>> covered that (I guess it doesn't). What you need
to do first is put all of
>>>>>>>>>>> your compiled code into a jar file.  Then, you'll
be utilizing the hadoop
>>>>>>>>>>> option -libjars.  You also need to make sure
that you set the hadoop
>>>>>>>>>>> classpath to include the giraph jar you are using
as well as your own jar.
>>>>>>>>>>> I actually asked a similar question not too long
ago; prior to this, I was
>>>>>>>>>>> just combining everything into one jar.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Your code should pretty much follow this format,
which is copied
>>>>>>>>>>> and paster from a previous response by Claudio:
>>>>>>>>>>>
>>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner
[ -D
>>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass
[
>>>>>>>>>>> GiraphRunner option e.g. -vip -vif etc. ]*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium
<
>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank You for your response. I could run
all the example
>>>>>>>>>>>> programs. I want to place my development
code in a different directory
>>>>>>>>>>>> other than the giraph directory. I copied
the
>>>>>>>>>>>> SimplePageRankComputation.java to a different
directory, compiled it(with a
>>>>>>>>>>>> different package name). But I couldnt replace
the "org.apache.
>>>>>>>>>>>> giraph.examples.SimpleShortestPathsComputation"
with my class
>>>>>>>>>>>> file. It gives me a ClassNotFoundException.
Please Guide me so that I can
>>>>>>>>>>>> run any class file with all required methods
(GIven the location of that
>>>>>>>>>>>> file).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando
<
>>>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> A "Quick Start" guide was recently added
to the Giraph
>>>>>>>>>>>>> website. Now learning Giraph is easier
than ever!  Here:
>>>>>>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek
Sembium <
>>>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am a new user in giraph. I could
run all example programs
>>>>>>>>>>>>>> in Giraph. Can someone please guide
me on how to run my own java program in
>>>>>>>>>>>>>> giraph (Assuming I have my input
file in jason format in HDFS).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>>>> University of Maryland
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>> University of Maryland
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>    Claudio Martella
>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>    Claudio Martella
>>>>>>    claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>    Claudio Martella
>>>>    claudio.martella@gmail.com
>>>>
>>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>>
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message