giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyoti Yadav <rao.jyoti26ya...@gmail.com>
Subject Re: Sample data for Single Source shortest path
Date Sat, 01 Mar 2014 08:58:48 GMT
Hi Sebastian..
Thanks for the links given  for big graphs..

Actually I want to tell you something about problem i am facing.

Initially I was working with *hadoop 0.20.203* . I build Giraph there.. it
was running fine.

Now  to test very big graph related problem and to compare the performance
, I moved to new system which is  of 64 cores and 512 GB memory and  3 TB
storage.  Instead to building Giraph in the new system, I just copied
Giraph folder from my previous system to this new system. In this new
system *hadoop version 2.2..0 * . I tried to execute
SimpleSourceShortestPath algo on sample data set. It is throwing following
exception.

I gave following command to execute the job.

hadoop jar
/home/abcd2014/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner -Dgiraph.SplitMasterWorker=true
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/abcd2014/giraph_input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/abcd2014/output2/shortestpaths -w 1



14/03/01 12:44:46 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
14/03/01 12:44:46 INFO utils.ConfigurationUtils: No edge output format
specified. Ensure your OutputFormat does not require one.
14/03/01 12:44:46 INFO Configuration.deprecation:
mapreduce.job.counters.limit is deprecated. Instead, use
mapreduce.job.counters.max
14/03/01 12:44:46 INFO Configuration.deprecation: mapred.job.map.memory.mb
is deprecated. Instead, use mapreduce.map.memory.mb
14/03/01 12:44:46 INFO Configuration.deprecation:
mapred.job.reduce.memory.mb is deprecated. Instead, use
mapreduce.reduce.memory.mb
14/03/01 12:44:46 INFO Configuration.deprecation:
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
14/03/01 12:44:46 INFO Configuration.deprecation:
mapreduce.user.classpath.first is deprecated. Instead, use
mapreduce.job.user.classpath.first
14/03/01 12:44:46 INFO Configuration.deprecation: mapred.map.max.attempts
is deprecated. Instead, use mapreduce.map.maxattempts
14/03/01 12:44:46 INFO job.GiraphJob: run: Since checkpointing is disabled
(default), do not allow any task retries (setting mapred.map.max.attempts =
0, old value = 4)
14/03/01 12:44:46 INFO Configuration.deprecation: mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address

*Exception in thread "main" java.lang.IllegalArgumentException:
checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run
in split master / worker mode since there is only 1 task at a time!*    at
org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(GiraphJob.java:165)
    at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:233)
    at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)



Would you suggest me something to fix this...If you need any details
further,please let me know...

Thanks & Regards

Jyoti




On Sat, Mar 1, 2014 at 1:35 PM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi Jyoti,
>
> You can find a couple of very large graphs in KONECT [1] and on the
> website of the laboratory for web algorithmics from the University of Milan
> [2]. You will probably have to convert them to an appropriate format for
> Giraph.
>
> Best,
> Sebastian
>
> [1] http://konect.uni-koblenz.de/
> [2] http://law.di.unimi.it/datasets.php
>
>
> On 03/01/2014 05:22 AM, Jyoti Yadav wrote:
>
>> Hi folks..
>>
>> I got new system which is  of 64 cores and 512 GB memory and  3 TB
>> storage.I want to test the performance of Giraph on this system.
>>   Would anyone provide me the link for very large graph  so that I can
>> execute Single Source Shortest Path Example. For this algo to run graph
>> should be weighted graph. and  to feed it into the Giraph -input format is
>> JsonLongDoubleFloatDouble
>>
>> Thanks in advance...
>> With Regards
>>
>> Jyoti
>>
>>
>

Mime
View raw message