giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Help needed for Running my own java programs in Giraph
Date Sun, 25 Aug 2013 13:03:36 GMT
you have this problem when you use two jars (one with giraph and one with
your classes) instead of a single fat-jar, correct? I tracked the same
problem a few weeks ago, basically zookeeper is run passing the wrong jar.


On Sat, Aug 24, 2013 at 4:51 PM, Vivek Sembium <vivek.sembium@gmail.com>wrote:

> Thank you for your suggestion. It worked. Its not giving class not found
> exception. But its giving me a new error
> Its stopping at map 0% and reduce 0%. Upon inspection I found that its
> unable to connect to zookeeper service.
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception
> onlineZooKeeperServers: Failed to connect in 10 tries!
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.IllegalStateException: onlineZooKeeperServers: Failed
> to connect in 10 tries!
>  at
> org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:727)
> at
> org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
>  at
> org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
>  at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
> ... 7 more
>
> Immediately I ran page rank benchmark and it executed successfully both
> from giraph in lib directory and also from giraphs own directory.
>
> Can you give me a very simple java program(finding maximum in a graph or
> simple page rank program) in giraph along with its jar file and input files
>  which I can place in my lib directory of hadoop and test if its working.
> And also the command to execute it. This should be added in the
> documentation as new comers can quickly setup giraph and concentrate on
> their project.
>
>
> On Sat, Aug 24, 2013 at 7:12 PM, Ahmet Emre Aladağ <emre.aladag@agmlab.com
> > wrote:
>
>> It isn't asking for edge input. It says make sure you don't need it. A
>> warning for the case you may have forgotten to give edge input when you
>> really needed.
>>
>> The cause of your error is what I'm wondering nowadays. I'm having a
>> similar problem. Currently I'm using a workaround: put all the jars
>> (giraph-core and my module giraph-nutch) in the lib folder of hadoop. Then
>> it works. But there should be a clean way of doing this.
>>
>> I should be able to say hadoop jar fat.jar ...
>>
>> Any help appreciated.
>>
>>
>>
>>
>>
>> ------------------------------
>> *Kimden: *"Vivek Sembium" <vivek.sembium@gmail.com>
>> *Kime: *user@giraph.apache.org
>> *Gönderilenler: *24 Ağustos Cumartesi 2013 11:51:49
>> *Konu: *Re: Help needed for Running my own java programs in Giraph
>>
>>
>> I tried with and without exporting hadoop classpath. I get the same error.
>>
>> Here's the command that I tried
>> hadoop jar
>> /mnt/a1/sda4/hadoop/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner -libjars
>> /mnt/a99/d0/vivek/workspace/Giraph/bin/SimplePageRankComputation.jar
>> practice.SimplePageRankComputation\$SimplePageRankMasterCompute -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/vivek/tiny_graph.txt -w 3
>>
>>
>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 13/08/24 14:21:00 INFO utils.ConfigurationUtils: No output format
>> specified. Ensure your OutputFormat does not require one.
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/giraph/master/DefaultMasterCompute
>>         at java.lang.ClassLoader.defineClass1(Native Method)
>>         at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
>>         at
>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>         at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
>>         at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:314)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>         at java.lang.Class.forName0(Native Method)
>>         at java.lang.Class.forName(Class.java:186)
>>         at
>> org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:425)
>>         at
>> org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:408)
>>         at
>> org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:200)
>>         at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>         at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:616)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.giraph.master.DefaultMasterCompute
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>         ... 26 more
>>
>>
>> I gave the graph input as JSON file. Why does it ask for edge input file
>> again?
>>
>>
>> Why isnt this working?
>>
>>
>> On Sat, Aug 24, 2013 at 12:35 AM, Kyle Orlando <kyle.r.orlando@gmail.com>wrote:
>>
>>> Hi Vivek,
>>>
>>> Sorry for the hasty response, I assumed the quick start guide covered
>>> that (I guess it doesn't). What you need to do first is put all of your
>>> compiled code into a jar file.  Then, you'll be utilizing the hadoop option
>>> -libjars.  You also need to make sure that you set the hadoop classpath to
>>> include the giraph jar you are using as well as your own jar. I actually
>>> asked a similar question not too long ago; prior to this, I was just
>>> combining everything into one jar.
>>>
>>>
>>> Your code should pretty much follow this format, which is copied and
>>> paster from a previous response by Claudio:
>>>
>>> export HADOOP_CLASSPATH="giraph.jar:your.jar"
>>> hadoop jar giraph.jar org.apache.giraph.GiraphRunner [ -D option ]*
>>> -libjars your.jar your.package.ComputationClass [ GiraphRunner option
>>> e.g. -vip -vif etc. ]*
>>>
>>>
>>> On Fri, Aug 23, 2013 at 2:51 PM, Vivek Sembium <vivek.sembium@gmail.com>wrote:
>>>
>>>> Thank You for your response. I could run all the example programs. I
>>>> want to place my development code in a different directory other than the
>>>> giraph directory. I copied the SimplePageRankComputation.java to a
>>>> different directory, compiled it(with a different package name). But I
>>>> couldnt replace the "org.apache.giraph.examples.SimpleShortestPathsComputation"
>>>> with my class file. It gives me a ClassNotFoundException. Please Guide me
>>>> so that I can run any class file with all required methods (GIven the
>>>> location of that file).
>>>>
>>>>
>>>> On Sat, Aug 24, 2013 at 12:07 AM, Kyle Orlando <
>>>> kyle.r.orlando@gmail.com> wrote:
>>>>
>>>>> A "Quick Start" guide was recently added to the Giraph website. Now
>>>>> learning Giraph is easier than ever!  Here:
>>>>> http://giraph.apache.org/quick_start.html
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Aug 23, 2013 at 2:34 PM, Vivek Sembium <
>>>>> vivek.sembium@gmail.com> wrote:
>>>>>
>>>>>> I am a new user in giraph. I could run all example programs in
>>>>>> Giraph. Can someone please guide me on how to run my own java program
in
>>>>>> giraph (Assuming I have my input file in jason format in HDFS).
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kyle Orlando
>>>>> Computer Engineering Major
>>>>> University of Maryland
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Kyle Orlando
>>> Computer Engineering Major
>>> University of Maryland
>>>
>>
>>
>>
>


-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
View raw message