giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Code failing for the large data
Date Thu, 10 Jan 2013 17:22:09 GMT
This looks like 0.1 (still using Hadoop RPC).  Please try trunk instead.

Avery

On 1/10/13 1:09 AM, pankaj Gulhane wrote:
> Hi,
>
> My code is working on smaller (very very small) dataset but if I use 
> the same code on the large dataset it fails.
>
> Following code is some basic implementation of naive PageRank (just 
> for testing). When I run with 4-5 vertices it works properly but when 
> run for thousands of vertices it fails with the following error
>
> <error>
> java.lang.IllegalStateException: run: Caught an unrecoverable 
> exception setup: Offlining servers due to exception...
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:668)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
>         at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.RuntimeException: setup: Offlining servers due to 
> exception...
>         at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>         ... 7 more
> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>         at 
> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582)
>         at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:458)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at 
> org.apache.giraph.comm.BasicRPCCommunications.sendPartitionReq(BasicRPCCommunications.java:817)
>         at 
> org.apache.giraph.graph.BspServiceWorker.loadVertices(BspServiceWorker.java:304)
>         at 
> org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:575)
>         ... 9 more
> </error>
>
>
> <code>
> public class PageRank implements Tool{
>     /** Configuration from Configurable */
>     private Configuration conf;
>     public static String SUPERSTEP_COUNT = 
> "PageRankBenchmark.superstepCount";
>
>     public static class PageRankHashMapVertex extends HashMapVertex<
>             LongWritable, DoubleWritable, DoubleWritable, 
> DoubleWritable> {
>
>
>         @Override
>         public void compute(Iterator<DoubleWritable> msgIterator) {
>
>             if (getSuperstep() >= 1) {
>                 double sum = 0;
>                 while (msgIterator.hasNext()) {
>                     sum += msgIterator.next().get();
>                 }
>                 DoubleWritable vertexValue =
>                         new DoubleWritable((0.15f / getNumVertices()) 
> + 0.85f *
>                         sum);
>                 setVertexValue(vertexValue);
>             }
>
>             if (getSuperstep() < getConf().getInt(SUPERSTEP_COUNT,4)) {
>                 long edges = getNumOutEdges();
>                 sendMsgToAllEdges(
>                         new DoubleWritable(getVertexValue().get() / 
> edges));
>             }
>
>             voteToHalt();
>         }
>     }
>
>     @Override
>     public Configuration getConf() {
>         return conf;
>     }
>
>     @Override
>     public void setConf(Configuration conf) {
>         this.conf = conf;
>     }
>
>     @Override
>     public int run(String[] args) throws Exception {
>         GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>
>         // job.setJarByClass(getClass());
>         job.setVertexClass(PageRankHashMapVertex.class);
>
> job.setVertexInputFormatClass(LongDoubleDoubleAdjacencyListVertexInputFormat.class);
> job.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);
>
>         job.setWorkerConfiguration(200, 200, 100.0f);
>         job.setJobName("Testing PG");
>
>         job.getConfiguration().setInt(SUPERSTEP_COUNT, 2);
>
>
>         FileInputFormat.addInputPath(job, new Path(args[0]));
>         FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>         return (job.run(true) == true ? 0 : 1);
>     }
>
>     public static void main(String[] args) throws Exception {
>         System.exit(ToolRunner.run(new PageRank(), args));
>     }
> }
>
> </code>
>
> Any pointers/help on the mistake I may be doing  would be great?
>
> Thanks,
> Pankaj
>
> PS: I am running on a cluster with more than 400 mapper slots.


Mime
View raw message