giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Wed, 28 Aug 2013 20:09:03 GMT
if you check out GIRAPH-750 [1], you will find a patch that fixes the
problem for now.
You'll have to apply the patch to trunk.

Best,
Claudio

[1] https://issues.apache.org/jira/browse/GIRAPH-750


On Mon, Aug 26, 2013 at 11:37 PM, Claudio Martella <
claudio.martella@gmail.com> wrote:

> yeah. i tracked the problem to what i mentioned earlier. ZK is run with
> the wrong jar when using -libjars. I have to figure out what's the expected
> behavior though, because the logic is kind obscure in the code.
>
>
> On Mon, Aug 26, 2013 at 11:24 PM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:
>
>> Yeah, exactly. We couldn't get it to work otherwise.
>>
>>
>> On Mon, Aug 26, 2013 at 11:00 AM, Claudio Martella <
>> claudio.martella@gmail.com> wrote:
>>
>>> you mean by running zookeeper independently?
>>>
>>>
>>> On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:
>>>
>>>> We were also experiencing similar problems when specifying -libjars as
>>>> opposed to just using a fat jar.  I believe we fixed it by setting the
>>>> giraph.zkList property, but this only appears to work when we list one node
>>>> as a zookeeper.
>>>>
>>>>
>>>> On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella <
>>>> claudio.martella@gmail.com> wrote:
>>>>
>>>>> OK, then I'm going to open an issue for that.
>>>>>
>>>>>
>>>>> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium <
>>>>> vivek.sembium@gmail.com> wrote:
>>>>>
>>>>>> Yes for the zookeeper problem I passed a seperate jar through
>>>>>> -libjars command. If I use additional jars zookeeper fails.
>>>>>>  On Aug 26, 2013 2:51 PM, "Claudio Martella" <
>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>
>>>>>>> there must be a misunderstanding. i was referring to the zookeeper
>>>>>>> problem.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <
>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>
>>>>>>>> No. I added my files(it was just a copy of one of the example
>>>>>>>> program to a different package) to the jar files of giraph.
But it was
>>>>>>>> still giving me classNotFoundException. Can you give me some
simple example
>>>>>>>> program with instructions on how to deploy it. So I can start
playing with
>>>>>>>> giraph and make changes to the program and learn, then start
working on my
>>>>>>>> project in giraph.
>>>>>>>> I will be very thankful if you can help me with this.
>>>>>>>>
>>>>>>>> Thanking you
>>>>>>>> -Vivek Sembium
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
>>>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> but you were still using an additional jar added through
-libjars,
>>>>>>>>> right?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <
>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> @Claudio Martella Your solution didnt work either.
I basically
>>>>>>>>>> tried copying the pageRankBenchmark to my own package,
renamed the package.
>>>>>>>>>> It compiles fine with giraph. But I couldnt run it
even if I add those
>>>>>>>>>> files to giraph jar before deployment. Help?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella
<
>>>>>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> you have this problem when you use two jars (one
with giraph and
>>>>>>>>>>> one with your classes) instead of a single fat-jar,
correct? I tracked the
>>>>>>>>>>> same problem a few weeks ago, basically zookeeper
is run passing the wrong
>>>>>>>>>>> jar.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium
<
>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your suggestion. It worked.
Its not giving class
>>>>>>>>>>>> not found exception. But its giving me a
new error
>>>>>>>>>>>> Its stopping at map 0% and reduce 0%. Upon
inspection I found
>>>>>>>>>>>> that its unable to connect to zookeeper service.
>>>>>>>>>>>>
>>>>>>>>>>>> java.lang.IllegalStateException: run: Caught
an unrecoverable
>>>>>>>>>>>> exception onlineZooKeeperServers: Failed
to connect in 10 tries!
>>>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>>>>>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>>>>>>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>>>>>>>> Caused by: java.lang.IllegalStateException:
>>>>>>>>>>>> onlineZooKeeperServers: Failed to connect
in 10 tries!
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>>>>>>>>>  at
>>>>>>>>>>>> org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>>>>>>>>>> ... 7 more
>>>>>>>>>>>>
>>>>>>>>>>>> Immediately I ran page rank benchmark and
it executed
>>>>>>>>>>>> successfully both from giraph in lib directory
and also from giraphs own
>>>>>>>>>>>> directory.
>>>>>>>>>>>>
>>>>>>>>>>>> Can you give me a very simple java program(finding
maximum in a
>>>>>>>>>>>> graph or simple page rank program) in giraph
along with its jar file and
>>>>>>>>>>>> input files  which I can place in my lib
directory of hadoop and test if
>>>>>>>>>>>> its working. And also the command to execute
it. This should be added in
>>>>>>>>>>>> the documentation as new comers can quickly
setup giraph and concentrate on
>>>>>>>>>>>> their project.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre
Aladağ <
>>>>>>>>>>>> emre.aladag@agmlab.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> It isn't asking for edge input. It says
make sure you don't
>>>>>>>>>>>>> need it. A warning for the case you may
have forgotten to give edge input
>>>>>>>>>>>>> when you really needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The cause of your error is what I'm wondering
nowadays. I'm
>>>>>>>>>>>>> having a similar problem. Currently I'm
using a workaround: put all the
>>>>>>>>>>>>> jars (giraph-core and my module giraph-nutch)
in the lib folder of hadoop.
>>>>>>>>>>>>> Then it works. But there should be a
clean way of doing this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I should be able to say hadoop jar fat.jar
...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>>>>>>>>>> *Kime: *user@giraph.apache.org
>>>>>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi
2013 11:51:49
>>>>>>>>>>>>> *Konu: *Re: Help needed for Running my
own java programs in
>>>>>>>>>>>>> Giraph
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I tried with and without exporting hadoop
classpath. I get the
>>>>>>>>>>>>> same error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the command that I tried
>>>>>>>>>>>>> hadoop jar
>>>>>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>>>>>>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute
-vif
>>>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No edge input
>>>>>>>>>>>>> format specified. Ensure your InputFormat
does not require one.
>>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No output
>>>>>>>>>>>>> format specified. Ensure your OutputFormat
does not require one.
>>>>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>>>>>>>>>         at java.lang.ClassLoader.defineClass1(Native
Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>>>>         at java.lang.Class.forName0(Native
Method)
>>>>>>>>>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>>>>         at
>>>>>>>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>>>>         ... 26 more
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I gave the graph input as JSON file.
Why does it ask for edge
>>>>>>>>>>>>> input file again?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why isnt this working?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle
Orlando <
>>>>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Vivek,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for the hasty response, I assumed
the quick start guide
>>>>>>>>>>>>>> covered that (I guess it doesn't).
What you need to do first is put all of
>>>>>>>>>>>>>> your compiled code into a jar file.
 Then, you'll be utilizing the hadoop
>>>>>>>>>>>>>> option -libjars.  You also need to
make sure that you set the hadoop
>>>>>>>>>>>>>> classpath to include the giraph jar
you are using as well as your own jar.
>>>>>>>>>>>>>> I actually asked a similar question
not too long ago; prior to this, I was
>>>>>>>>>>>>>> just combining everything into one
jar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Your code should pretty much follow
this format, which is
>>>>>>>>>>>>>> copied and paster from a previous
response by Claudio:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner
[ -D
>>>>>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass
[
>>>>>>>>>>>>>> GiraphRunner option e.g. -vip -vif
etc. ]*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM,
Vivek Sembium <
>>>>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank You for your response.
I could run all the example
>>>>>>>>>>>>>>> programs. I want to place my
development code in a different directory
>>>>>>>>>>>>>>> other than the giraph directory.
I copied the
>>>>>>>>>>>>>>> SimplePageRankComputation.java
to a different directory, compiled it(with a
>>>>>>>>>>>>>>> different package name). But
I couldnt replace the "org.
>>>>>>>>>>>>>>> apache.giraph.examples.SimpleShortestPathsComputation"
with
>>>>>>>>>>>>>>> my class file. It gives me a
ClassNotFoundException. Please Guide me so
>>>>>>>>>>>>>>> that I can run any class file
with all required methods (GIven the location
>>>>>>>>>>>>>>> of that file).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07
AM, Kyle Orlando <
>>>>>>>>>>>>>>> kyle.r.orlando@gmail.com>
wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> A "Quick Start" guide was
recently added to the Giraph
>>>>>>>>>>>>>>>> website. Now learning Giraph
is easier than ever!  Here:
>>>>>>>>>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34
PM, Vivek Sembium <
>>>>>>>>>>>>>>>> vivek.sembium@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am a new user in giraph.
I could run all example
>>>>>>>>>>>>>>>>> programs in Giraph. Can
someone please guide me on how to run my own java
>>>>>>>>>>>>>>>>> program in giraph (Assuming
I have my input file in jason format in HDFS).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>>>>>>> University of Maryland
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>>>>> University of Maryland
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>    Claudio Martella
>>>>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>    Claudio Martella
>>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>    Claudio Martella
>>>>>>>    claudio.martella@gmail.com
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>    Claudio Martella
>>>>>    claudio.martella@gmail.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Kyle Orlando
>>>> Computer Engineering Major
>>>> University of Maryland
>>>>
>>>
>>>
>>>
>>> --
>>>    Claudio Martella
>>>    claudio.martella@gmail.com
>>>
>>
>>
>>
>> --
>> Kyle Orlando
>> Computer Engineering Major
>> University of Maryland
>>
>
>
>
> --
>    Claudio Martella
>    claudio.martella@gmail.com
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message