giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Castagna <castagna.li...@googlemail.com>
Subject Re: A simple use case: shortest paths on a FOAF (i.e. Friend of a Friend) graph
Date Fri, 13 Apr 2012 11:42:08 GMT
Paolo Castagna wrote:
> Avery Ching wrote:
>> It shouldn't be, your code looks very similar to the unittests (i.e.
>> TestManualCheckpoint.java).  So, you're trying to run your test with the
>> local hadoop (similar to the unittests)?  Or are you using an actual
>> hadoop setup?
> 
> Hi Avery,
> here is a few more details on what I am trying to do, in order to run my Giraph
> jobs on a local Hadoop running (for testing and debugging stuff locally):
> 
>   GiraphJob job = new GiraphJob("shortest paths");
>   Configuration conf = job.getConfiguration();
>   conf.setBoolean(GiraphJob.SPLIT_MASTER_WORKER, false);
>   conf.setBoolean(GiraphJob.LOCAL_TEST_MODE, true);
>   // conf.set(GiraphJob.ZOOKEEPER_JAR,
> "file://target/dependency/zookeeper-3.3.3.jar");
>   job.setWorkerConfiguration(1, 1, 100.0f);
>   job.setVertexClass(SimpleShortestPathsVertex.class);
>   job.setVertexInputFormatClass(SimpleShortestPathsVertexInputFormat.class);
>   job.setVertexOutputFormatClass(SimpleShortestPathsVertexOutputFormat.class);
>   FileInputFormat.addInputPath(job.getInternalJob(), new
> Path("src/main/resources/giraph1.txt"));
>   Path outputPath = new Path("target/giraph1");
>   FileSystem hdfs = FileSystem.get(conf);
>   hdfs.delete(outputPath, true);
>   FileOutputFormat.setOutputPath(job.getInternalJob(), outputPath);
>   job.run(true);
> 
> Am I doing something wrong/stupid here?
> Am I missing something important? (probably! but I do not see what I am missing)

This is a better way:

  Iterable<String> results = InternalVertexRunner.run(
    SimpleShortestPathsVertex.class,
    SimpleShortestPathsVertex.SimpleShortestPathsVertexInputFormat.class,
    SimpleShortestPathsVertex.SimpleShortestPathsVertexOutputFormat.class,
    params, graph);

... which starts a local ZooKeeper properly.

However, I still have a question: when I run it in a unit test everything is
fine. When I run it on a Java main method, it hangs towards the end.

Paolo

Mime
View raw message