giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Mon, 26 Aug 2013 21:37:03 GMT
yeah. i tracked the problem to what i mentioned earlier. ZK is run with the
wrong jar when using -libjars. I have to figure out what's the expected
behavior though, because the logic is kind obscure in the code.


On Mon, Aug 26, 2013 at 11:24 PM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:

> Yeah, exactly. We couldn't get it to work otherwise.
>
>
> On Mon, Aug 26, 2013 at 11:00 AM, Claudio Martella <
> claudio.martella@gmail.com> wrote:
>
>> you mean by running zookeeper independently?
>>
>>
>> On Mon, Aug 26, 2013 at 3:16 PM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:
>>
>>> We were also experiencing similar problems when specifying -libjars as
>>> opposed to just using a fat jar.  I believe we fixed it by setting the
>>> giraph.zkList property, but this only appears to work when we list one node
>>> as a zookeeper.
>>>
>>>
>>> On Mon, Aug 26, 2013 at 8:55 AM, Claudio Martella <
>>> claudio.martella@gmail.com> wrote:
>>>
>>>> OK, then I'm going to open an issue for that.
>>>>
>>>>
>>>> On Mon, Aug 26, 2013 at 11:23 AM, Vivek Sembium <
>>>> vivek.sembium@gmail.com> wrote:
>>>>
>>>>> Yes for the zookeeper problem I passed a seperate jar through -libjars
>>>>> command. If I use additional jars zookeeper fails.
>>>>>  On Aug 26, 2013 2:51 PM, "Claudio Martella" <
>>>>> claudio.martella@gmail.com> wrote:
>>>>>
>>>>>> there must be a misunderstanding. i was referring to the zookeeper
>>>>>> problem.
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 26, 2013 at 11:14 AM, Vivek Sembium <
>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>
>>>>>>> No. I added my files(it was just a copy of one of the example
>>>>>>> program to a different package) to the jar files of giraph. But
it was
>>>>>>> still giving me classNotFoundException. Can you give me some
simple example
>>>>>>> program with instructions on how to deploy it. So I can start
playing with
>>>>>>> giraph and make changes to the program and learn, then start
working on my
>>>>>>> project in giraph.
>>>>>>> I will be very thankful if you can help me with this.
>>>>>>>
>>>>>>> Thanking you
>>>>>>> -Vivek Sembium
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 26, 2013 at 2:37 PM, Claudio Martella <
>>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>>
>>>>>>>> but you were still using an additional jar added through
-libjars,
>>>>>>>> right?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 26, 2013 at 8:43 AM, Vivek Sembium <
>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> @Claudio Martella Your solution didnt work either. I
basically
>>>>>>>>> tried copying the pageRankBenchmark to my own package,
renamed the package.
>>>>>>>>> It compiles fine with giraph. But I couldnt run it even
if I add those
>>>>>>>>> files to giraph jar before deployment. Help?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Aug 25, 2013 at 6:33 PM, Claudio Martella <
>>>>>>>>> claudio.martella@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> you have this problem when you use two jars (one
with giraph and
>>>>>>>>>> one with your classes) instead of a single fat-jar,
correct? I tracked the
>>>>>>>>>> same problem a few weeks ago, basically zookeeper
is run passing the wrong
>>>>>>>>>> jar.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <
>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you for your suggestion. It worked. Its
not giving class
>>>>>>>>>>> not found exception. But its giving me a new
error
>>>>>>>>>>> Its stopping at map 0% and reduce 0%. Upon inspection
I found
>>>>>>>>>>> that its unable to connect to zookeeper service.
>>>>>>>>>>>
>>>>>>>>>>> java.lang.IllegalStateException: run: Caught
an unrecoverable
>>>>>>>>>>> exception onlineZooKeeperServers: Failed to connect
in 10 tries!
>>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>>>>>>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>>>>>>>>>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>>>  at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>>>>>>>>>>>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>>>>>>>> Caused by: java.lang.IllegalStateException:
>>>>>>>>>>> onlineZooKeeperServers: Failed to connect in
10 tries!
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>>>>>>>>>>>  at
>>>>>>>>>>> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
>>>>>>>>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>>>>>>>>>>>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
>>>>>>>>>>> ... 7 more
>>>>>>>>>>>
>>>>>>>>>>> Immediately I ran page rank benchmark and it
executed
>>>>>>>>>>> successfully both from giraph in lib directory
and also from giraphs own
>>>>>>>>>>> directory.
>>>>>>>>>>>
>>>>>>>>>>> Can you give me a very simple java program(finding
maximum in a
>>>>>>>>>>> graph or simple page rank program) in giraph
along with its jar file and
>>>>>>>>>>> input files  which I can place in my lib directory
of hadoop and test if
>>>>>>>>>>> its working. And also the command to execute
it. This should be added in
>>>>>>>>>>> the documentation as new comers can quickly setup
giraph and concentrate on
>>>>>>>>>>> their project.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ
<
>>>>>>>>>>> emre.aladag@agmlab.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> It isn't asking for edge input. It says make
sure you don't
>>>>>>>>>>>> need it. A warning for the case you may have
forgotten to give edge input
>>>>>>>>>>>> when you really needed.
>>>>>>>>>>>>
>>>>>>>>>>>> The cause of your error is what I'm wondering
nowadays. I'm
>>>>>>>>>>>> having a similar problem. Currently I'm using
a workaround: put all the
>>>>>>>>>>>> jars (giraph-core and my module giraph-nutch)
in the lib folder of hadoop.
>>>>>>>>>>>> Then it works. But there should be a clean
way of doing this.
>>>>>>>>>>>>
>>>>>>>>>>>> I should be able to say hadoop jar fat.jar
...
>>>>>>>>>>>>
>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>>>>>>>>>>>> *Kime: *user@giraph.apache.org
>>>>>>>>>>>> *Gönderilenler: *24 Ağustos Cumartesi 2013
11:51:49
>>>>>>>>>>>> *Konu: *Re: Help needed for Running my own
java programs in
>>>>>>>>>>>> Giraph
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I tried with and without exporting hadoop
classpath. I get the
>>>>>>>>>>>> same error.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the command that I tried
>>>>>>>>>>>> hadoop jar
>>>>>>>>>>>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>>>>>>>>>>>> org.apache.giraph.GiraphRunner -libjars
>>>>>>>>>>>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>>>>>>>>>>>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute
-vif
>>>>>>>>>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>>>>>>>>>> -vip /user/vivek/tiny_graph.txt -w 3
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No edge input
>>>>>>>>>>>> format specified. Ensure your InputFormat
does not require one.
>>>>>>>>>>>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils:
No output
>>>>>>>>>>>> format specified. Ensure your OutputFormat
does not require one.
>>>>>>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>>>>>> org/apache/giraph/master/DefaultMasterCompute
>>>>>>>>>>>>         at java.lang.ClassLoader.defineClass1(Native
Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>>>         at java.lang.Class.forName0(Native
Method)
>>>>>>>>>>>>         at java.lang.Class.forName(Class.java:186)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>>>>>>         at
>>>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>>>>>>>>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>> org.apache.giraph.master.DefaultMasterCompute
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>>>>>>>>>>>         at java.security.AccessController.doPrivileged(Native
>>>>>>>>>>>> Method)
>>>>>>>>>>>>         at
>>>>>>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>>>>>>>>>>>         at
>>>>>>>>>>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>>>>>>>>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>>>>>>>>>>>         ... 26 more
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I gave the graph input as JSON file. Why
does it ask for edge
>>>>>>>>>>>> input file again?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Why isnt this working?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando
<
>>>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Vivek,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry for the hasty response, I assumed
the quick start guide
>>>>>>>>>>>>> covered that (I guess it doesn't). What
you need to do first is put all of
>>>>>>>>>>>>> your compiled code into a jar file. 
Then, you'll be utilizing the hadoop
>>>>>>>>>>>>> option -libjars.  You also need to make
sure that you set the hadoop
>>>>>>>>>>>>> classpath to include the giraph jar you
are using as well as your own jar.
>>>>>>>>>>>>> I actually asked a similar question not
too long ago; prior to this, I was
>>>>>>>>>>>>> just combining everything into one jar.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your code should pretty much follow this
format, which is
>>>>>>>>>>>>> copied and paster from a previous response
by Claudio:
>>>>>>>>>>>>>
>>>>>>>>>>>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>>>>>>>>>>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner
[ -D
>>>>>>>>>>>>> option ]* -libjars your.jar your.package.ComputationClass
[
>>>>>>>>>>>>> GiraphRunner option e.g. -vip -vif etc.
]*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek
Sembium <
>>>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank You for your response. I could
run all the example
>>>>>>>>>>>>>> programs. I want to place my development
code in a different directory
>>>>>>>>>>>>>> other than the giraph directory.
I copied the
>>>>>>>>>>>>>> SimplePageRankComputation.java to
a different directory, compiled it(with a
>>>>>>>>>>>>>> different package name). But I couldnt
replace the "org.
>>>>>>>>>>>>>> apache.giraph.examples.SimpleShortestPathsComputation"
with
>>>>>>>>>>>>>> my class file. It gives me a ClassNotFoundException.
Please Guide me so
>>>>>>>>>>>>>> that I can run any class file with
all required methods (GIven the location
>>>>>>>>>>>>>> of that file).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Aug 24, 2013 at 12:07 AM,
Kyle Orlando <
>>>>>>>>>>>>>> kyle.r.orlando@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A "Quick Start" guide was recently
added to the Giraph
>>>>>>>>>>>>>>> website. Now learning Giraph
is easier than ever!  Here:
>>>>>>>>>>>>>>> http://giraph.apache.org/quick_start.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 23, 2013 at 2:34
PM, Vivek Sembium <
>>>>>>>>>>>>>>> vivek.sembium@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am a new user in giraph.
I could run all example programs
>>>>>>>>>>>>>>>> in Giraph. Can someone please
guide me on how to run my own java program in
>>>>>>>>>>>>>>>> giraph (Assuming I have my
input file in jason format in HDFS).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>>>>>> University of Maryland
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Kyle Orlando
>>>>>>>>>>>>> Computer Engineering Major
>>>>>>>>>>>>> University of Maryland
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>    Claudio Martella
>>>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>    Claudio Martella
>>>>>>>>    claudio.martella@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>    Claudio Martella
>>>>>>    claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>    Claudio Martella
>>>>    claudio.martella@gmail.com
>>>>
>>>
>>>
>>>
>>> --
>>> Kyle Orlando
>>> Computer Engineering Major
>>> University of Maryland
>>>
>>
>>
>>
>> --
>>    Claudio Martella
>>    claudio.martella@gmail.com
>>
>
>
>
> --
> Kyle Orlando
> Computer Engineering Major
> University of Maryland
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message