giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Mon, 26 Aug 2013 12:55:51 GMT
OK, then I'm going to open an issue for that.


On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:

> Yes for the zookeeper problem I passed a seperate jar through -libjars
> command. If I use additional jars zookeeper fails.
>  On Aug 26, 2013 2:51 PM, "Claudio Martella" <claudio.martella@gmail.com>
> wrote:
>
>> there must be a misunderstanding. i was referring to the zookeeper
>> problem.
>>
>>
>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>>
>>> No. I added my files(it was just a copy of one of the example program to
>>> a different package) to the jar files of giraph. But it was still giving me
>>> classNotFoundException. Can you give me some simple example program with
>>> instructions on how to deploy it. So I can start playing with giraph and
>>> make changes to the program and learn, then start working on my project in
>>> giraph.
>>> I will be very thankful if you can help me with this.
>>>
>>> Thanking you
>>> -Vivek Sembium
>>>
>>>
>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
>>> claudio.martella@gmail.com> wrote:
>>>
>>>> but you were still using an additional jar added through -libjars,
>>>> right?
>>>>
>>>>
>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <vivek.sembium@gmail.com
>>>> > wrote:
>>>>
>>>>> @Claudio Martella Your solution didnt work either. I basically tried
>>>>> copying the pageRankBenchmark to my own package, renamed the package.
It
>>>>> compiles fine with giraph. But I couldnt run it even if I add those files
>>>>> to giraph jar before deployment. Help?
>>>>>
>>>>>
>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella <
>>>>> claudio.martella@gmail.com> wrote:
>>>>>
>>>>>> you have this problem when you use two jars (one with giraph and
one
>>>>>> with your classes) instead of a single fat-jar, correct? I tracked
the same
>>>>>> problem a few weeks ago, basically zookeeper is run passing the wrong
jar.
>>>>>>
>>>>>>
>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <
>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>
>>>>>>> Thank you for your suggestion. It worked. Its not giving class
not
>>>>>>> found exception. But its giving me a new error
>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection I found
that
>>>>>>> its unable to connect to zookeeper service.
>>>>>>>
>>>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>>>>> exception onlineZooKeeperServers: Failed to connect in 10 tries!
>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>>> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers:
>>>>>>> Failed to connect in 10 tries!
>>>>>>>  at
>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>>>>> at
>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>>>>  at
>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>>>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>>>>> ... 7 more
>>>>>>>
>>>>>>> Immediately I ran page rank benchmark and it executed successfully
>>>>>>> both from giraph in lib directory and also from giraphs own directory.
>>>>>>>
>>>>>>> Can you give me a very simple java program(finding maximum in
a
>>>>>>> graph or simple page rank program) in giraph along with its jar
file and
>>>>>>> input files  which I can place in my lib directory of hadoop
and test if
>>>>>>> its working. And also the command to execute it. This should
be added in
>>>>>>> the documentation as new comers can quickly setup giraph and
concentrate on
>>>>>>> their project.
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ <
>>>>>>> emre.aladag@agmlab.com> wrote:
>>>>>>>
>>>>>>>> It isn't asking for edge input. It says make sure you don't
need
>>>>>>>> it. A warning for the case you may have forgotten to give
edge input when
>>>>>>>> you really needed.
>>>>>>>>
>>>>>>>> The cause of your error is what I'm wondering nowadays. I'm
having
>>>>>>>> a similar problem. Currently I'm using a workaround: put
all the jars
>>>>>>>> (giraph-core and my module giraph-nutch) in the lib folder
of hadoop. Then
>>>>>>>> it works. But there should be a clean way of doing this.
>>>>>>>>
>>>>>>>> I should be able to say hadoop jar fat.jar ...
>>>>>>>>
>>>>>>>> Any help appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>>>>> *Kime: *user@giraph.apache.org
>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49
>>>>>>>> *Konu: *Re: Help needed for Running my own java programs
in Giraph
>>>>>>>>
>>>>>>>>
>>>>>>>> I tried with and without exporting hadoop classpath. I get
the same
>>>>>>>> error.
>>>>>>>>
>>>>>>>> Here's the command that I tried
>>>>>>>> hadoop jar
>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute
-vif
>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>>>>
>>>>>>>>
>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge
input
>>>>>>>> format specified. Ensure your InputFormat does not require
one.
>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output
format
>>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>>>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>>>>         at
>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>>         at
>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>>>>         at
>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>> Method)
>>>>>>>>         at
>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>         at
>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>         at java.lang.Class.forName0(Native Method)
>>>>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>>>>         at
>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>>>>         at
>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>>>>         at
>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>>>>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>         at
>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>> Method)
>>>>>>>>         at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>         at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>> Method)
>>>>>>>>         at
>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>         at
>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>         ... 26 more
>>>>>>>>
>>>>>>>>
>>>>>>>> I gave the graph input as JSON file. Why does it ask for
edge input
>>>>>>>> file again?
>>>>>>>>
>>>>>>>>
>>>>>>>> Why isnt this working?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando <
>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Vivek,
>>>>>>>>>
>>>>>>>>> Sorry for the hasty response, I assumed the quick start
guide
>>>>>>>>> covered that (I guess it doesn't). What you need to do
first is put all of
>>>>>>>>> your compiled code into a jar file.  Then, you'll be
utilizing the hadoop
>>>>>>>>> option -libjars.  You also need to make sure that you
set the hadoop
>>>>>>>>> classpath to include the giraph jar you are using as
well as your own jar.
>>>>>>>>> I actually asked a similar question not too long ago;
prior to this, I was
>>>>>>>>> just combining everything into one jar.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Your code should pretty much follow this format, which
is copied
>>>>>>>>> and paster from a previous response by Claudio:
>>>>>>>>>
>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner
[ -D option
>>>>>>>>> ]* -libjars your.jar your.package.ComputationClass [
GiraphRunner
>>>>>>>>> option e.g. -vip -vif etc. ]*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium <
>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thank You for your response. I could run all the
example
>>>>>>>>>> programs. I want to place my development code in
a different directory
>>>>>>>>>> other than the giraph directory. I copied the
>>>>>>>>>> SimplePageRankComputation.java to a different directory,
compiled it(with a
>>>>>>>>>> different package name). But I couldnt replace the
"org.apache.
>>>>>>>>>> giraph.examples.SimpleShortestPathsComputation" with
my class
>>>>>>>>>> file. It gives me a ClassNotFoundException. Please
Guide me so that I can
>>>>>>>>>> run any class file with all required methods (GIven
the location of that
>>>>>>>>>> file).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando <
>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> A "Quick Start" guide was recently added to the
Giraph website.
>>>>>>>>>>> Now learning Giraph is easier than ever!  Here:
>>>>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium
<
>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I am a new user in giraph. I could run all
example programs in
>>>>>>>>>>>> Giraph. Can someone please guide me on how
to run my own java program in
>>>>>>>>>>>> giraph (Assuming I have my input file in
jason format in HDFS).
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>> University of Maryland
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Kyle Orlando
>>>>>>>>> Computer Engineering Major
>>>>>>>>> University of Maryland
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>    Claudio Martella
>>>>>>    claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>    Claudio Martella
>>>>    claudio.martella@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>>
>


-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message