giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paolo Castagna <castagna.li...@googlemail.com>
Subject Re: A simple use case: shortest paths on a FOAF (i.e. Friend of a Friend) graph
Date Fri, 13 Apr 2012 09:20:42 GMT
Avery Ching wrote:
> It shouldn't be, your code looks very similar to the unittests (i.e.
> TestManualCheckpoint.java).  So, you're trying to run your test with the
> local hadoop (similar to the unittests)?  Or are you using an actual
> hadoop setup?

Hi Avery,
here is a few more details on what I am trying to do, in order to run my Giraph
jobs on a local Hadoop running (for testing and debugging stuff locally):

  GiraphJob job = new GiraphJob("shortest paths");
  Configuration conf = job.getConfiguration();
  conf.setBoolean(GiraphJob.SPLIT_MASTER_WORKER, false);
  conf.setBoolean(GiraphJob.LOCAL_TEST_MODE, true);
  // conf.set(GiraphJob.ZOOKEEPER_JAR,
"file://target/dependency/zookeeper-3.3.3.jar");
  job.setWorkerConfiguration(1, 1, 100.0f);
  job.setVertexClass(SimpleShortestPathsVertex.class);
  job.setVertexInputFormatClass(SimpleShortestPathsVertexInputFormat.class);
  job.setVertexOutputFormatClass(SimpleShortestPathsVertexOutputFormat.class);
  FileInputFormat.addInputPath(job.getInternalJob(), new
Path("src/main/resources/giraph1.txt"));
  Path outputPath = new Path("target/giraph1");
  FileSystem hdfs = FileSystem.get(conf);
  hdfs.delete(outputPath, true);
  FileOutputFormat.setOutputPath(job.getInternalJob(), outputPath);
  job.run(true);

Am I doing something wrong/stupid here?
Am I missing something important? (probably! but I do not see what I am missing)

This is what I think happens...

In GraphMapper something goes wrong during setup(context), probably because
GiraphJob.ZOOKEEPER_JAR is not set(?) and an exception different from
IOException is thrown and I do not see any useful error message:

    try {
      setup(context);
      while (context.nextKeyValue()) {
        map(context.getCurrentKey(),
            context.getCurrentValue(),
            context);
      }
      cleanup(context);
    } catch (IOException e) {
      if (mapFunctions == MapFunctions.WORKER_ONLY) {
        serviceWorker.failureCleanup();
      }
      throw new IllegalStateException(
          "run: Caught an unrecoverable exception " + e.getMessage(), e);
    }

My question is: is it possible to run a Giraph job as I am trying to do above
(for testing only) or developers need to have an Hadoop cluster either remote
or locally and ZooKeeper running (either remote or locally)?

Thanks,
Paolo

Mime
View raw message