giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Compton <compton.r...@gmail.com>
Subject Re: Where can I find a simple "Hello World" example for Giraph
Date Fri, 22 Feb 2013 00:10:33 GMT
Ok, I've been looking at the PageRankBenchmark. There's a lot going on
in there...

It looks like the minimum amount of stuff I need to run a do-nothing
job is what I've got below.

But now it's telling me that (and PageRankBenchmark doesn't have the
word "output" anywhere).

13/02/21 16:08:37 ERROR security.UserGroupInformation:
PriviledgedActionException as:rfcompton (auth:SIMPLE)
cause:org.apache.hadoop.mapred.InvalidJobConfException: Output
directory not set.
Exception in thread "main"
org.apache.hadoop.mapred.InvalidJobConfException: Output directory not
set.
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:123)
at org.apache.giraph.io.formats.TextVertexOutputFormat.checkOutputSpecs(TextVertexOutputFormat.java:55)
at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:48)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
at org.apache.giraph.graph.GiraphJob.run(GiraphJob.java:282)
at com.hrl.issl.osi.networks.HelloGiraph0p2.run(HelloGiraph0p2.java:49)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.hrl.issl.osi.networks.HelloGiraph0p2.main(HelloGiraph0p2.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)



import org.apache.giraph.conf.GiraphConfiguration;
import org.apache.giraph.graph.GiraphJob;
import org.apache.giraph.io.formats.JsonBase64VertexInputFormat;
import org.apache.giraph.io.formats.JsonBase64VertexOutputFormat;
import org.apache.giraph.io.formats.TextVertexInputFormat;
import org.apache.giraph.io.formats.TextVertexOutputFormat;
import org.apache.giraph.vertex.SimpleVertex;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

/**
 *
 * Hello world giraph 0.2...
 *
 */
public class HelloGiraph0p2 implements Tool {
/** Configuration */
private Configuration conf;

@Override
public void setConf(Configuration conf) {
this.conf = conf;
}

@Override
public Configuration getConf() {
return conf;
}

@Override
public int run(String[] arg0) throws Exception {

GiraphJob job = new GiraphJob(getConf(), getClass().getName());

GiraphConfiguration configuration = job.getConfiguration();

configuration.setVertexClass(SimpleVertex.class);

configuration.setVertexInputFormatClass(JsonBase64VertexInputFormat.class);
configuration.setVertexOutputFormatClass(JsonBase64VertexOutputFormat.class);

configuration.setWorkerConfiguration(30, 30, 100.0f);

return job.run(true) ? 0 : -1;

}

public static void main(String[] args) throws Exception {

System.exit(ToolRunner.run(new HelloGiraph0p2(), args));
}

}

On Thu, Feb 21, 2013 at 2:58 PM, Maja Kabiljo <majakabiljo@fb.com> wrote:
> Hi Ryan,
>
> Before running the job, you need to set Vertex and input/output format
> classes on it. Please take a look at one of the benchmarks to see how to
> do that. Alternatively, you can try using GiraphRunner, where you pass
> these classes as command line arguments.
>
> Maja
>
> On 2/21/13 2:43 PM, "Ryan Compton" <compton.ryan@gmail.com> wrote:
>
>>I'm still struggling with this. I am trying to use 0.2, I dont have
>>permissions to edit core-site.xml
>>
>>I think this the most basic boiler plate code for a 0.2 Giraph
>>project, but I still can't run it.
>>
>>Exception in thread "main" java.lang.NullPointerException
>>at
>>org.apache.giraph.utils.ReflectionUtils.getTypeArguments(ReflectionUtils.j
>>ava:85)
>>at
>>org.apache.giraph.conf.GiraphClasses.readFromConf(GiraphClasses.java:117)
>>at org.apache.giraph.conf.GiraphClasses.<init>(GiraphClasses.java:105)
>>at
>>org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.<init>(Immutabl
>>eClassesGiraphConfiguration.java:84)
>>at
>>com.hrl.issl.osi.networks.HelloGiraph0p2.setConf(HelloGiraph0p2.java:34)
>>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:61)
>>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>at com.hrl.issl.osi.networks.HelloGiraph0p2.main(HelloGiraph0p2.java:70)
>>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>39)
>>at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>pl.java:25)
>>at java.lang.reflect.Method.invoke(Method.java:597)
>>at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>>
>>
>>
>>package networks;
>>
>>import java.io.IOException;
>>
>>import org.apache.giraph.conf.ImmutableClassesGiraphConfiguration;
>>import org.apache.giraph.graph.GiraphJob;
>>import org.apache.giraph.vertex.EdgeListVertex;
>>import org.apache.hadoop.conf.Configuration;
>>import org.apache.hadoop.io.LongWritable;
>>import org.apache.hadoop.io.Text;
>>import org.apache.hadoop.util.Tool;
>>import org.apache.hadoop.util.ToolRunner;
>>import org.apache.log4j.Logger;
>>
>>/**
>> *
>> * Hello world giraph 0.2...
>> *
>> */
>>public class HelloGiraph0p2 extends EdgeListVertex<LongWritable, Text,
>>Text, Text> implements Tool {
>>/** Configuration */
>>private ImmutableClassesGiraphConfiguration<LongWritable, Text, Text,
>>Text> conf;
>>/** Class logger */
>>private static final Logger LOG = Logger.getLogger(HelloGiraph0p2.class);
>>
>>@Override
>>public void compute(Iterable<Text> arg0) throws IOException {
>>int four = 2+2;
>>}
>>@Override
>>public void setConf(Configuration configurationIn) {
>>this.conf = new ImmutableClassesGiraphConfiguration<LongWritable,
>>Text, Text, Text>(configurationIn);
>>return;
>>}
>>@Override
>>public ImmutableClassesGiraphConfiguration<LongWritable, Text, Text,
>>Text> getConf() {
>>return conf;
>>}
>>
>>/**
>>*
>>* ToolRunner run
>>*
>>* @param arg0
>>* @return
>>* @throws Exception
>>*/
>>@Override
>>public int run(String[] arg0) throws Exception {
>>GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>>
>>return job.run(true) ? 0 : -1;
>>
>>}
>>/**
>>* main...
>>*
>>* @param args
>>* @throws Exception
>>*/
>>public static void main(String[] args) throws Exception {
>>System.exit(ToolRunner.run(new HelloGiraph0p2(), args));
>>}
>>
>>}
>>
>>
>>
>>On Tue, Feb 5, 2013 at 4:24 AM, Gustavo Enrique Salazar Torres
>><gsalazar@ime.usp.br> wrote:
>>> Hi Ryan:
>>>
>>> I got that same error and discovered that I have to start a zookeeper
>>> instance. What I did was to download Zookeeper, write a new zoo.cfg file
>>> under conf directory with the following:
>>>
>>> dataDir=/home/user/zookeeper-3.4.5/tmp
>>> clientPort=2181
>>>
>>> Also I added some lines in Hadoop's core-site.xml:
>>> <property>
>>>     <name>giraph.zkList</name>
>>>     <value>localhost:2181</value>
>>>   </property>
>>>
>>> Then I start Zookeper with bin/zkServer.sh start (also you will have to
>>> restart Hadoop) and then you can launch your Giraph Job.
>>> This setup worked for me (maybe there is an easiest way :D), hope it is
>>> useful.
>>>
>>> Best regards
>>> Gustavo
>>>
>>>
>>> On Mon, Feb 4, 2013 at 10:06 PM, Ryan Compton <compton.ryan@gmail.com>
>>> wrote:
>>>>
>>>> Ok great, thanks. I've been working with 0.1, I can get things to
>>>> compile (see below code) but they still are not running, the maps hang
>>>> (also below). I have no idea how to fix it, I may consider updating
>>>> that code I have that compiles to 0.2 and see if it works then. The
>>>> only difference I can see is that 0.2 requires everything have a
>>>> "message"
>>>>
>>>> -bash-3.2$ hadoop jar target/giraph-0.1-jar-with-dependencies.jar
>>>> com.SimpleGiraphSumEdgeWeights /user/rfcompton/giraphTSPInput
>>>> /user/rfcompton/giraphTSPOutput 3 3
>>>> 13/02/04 15:48:23 INFO mapred.JobClient: Running job:
>>>> job_201301230932_1199
>>>> 13/02/04 15:48:24 INFO mapred.JobClient:  map 0% reduce 0%
>>>> 13/02/04 15:48:35 INFO mapred.JobClient:  map 25% reduce 0%
>>>> 13/02/04 15:58:40 INFO mapred.JobClient: Task Id :
>>>> attempt_201301230932_1199_m_000003_0, Status : FAILED
>>>> java.lang.IllegalStateException: run: Caught an unrecoverable
>>>> exception setup: Offlining servers due to exception...
>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:641)
>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>>
>>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>>>>n.java:1157)
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>>>> Caused by: java.lang.RuntimeException: setup: Offlining servers due to
>>>> exception...
>>>> at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:466)
>>>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:630)
>>>> ... 7 more
>>>> Caused by: java.lang.IllegalStateException: setup: loadVertices failed
>>>> at
>>>>
>>>>org.apache.giraph.graph.BspServiceWorker.setup(BspServiceWorker.java:582
>>>>)
>>>> at org.apache.
>>>> Task attempt_201301230932_1199_m_000003_0 failed to report status for
>>>> 600 seconds. Killing!
>>>> 13/02/04 15:58:43 INFO mapred.JobClient: Task Id :
>>>> attempt_201301230932_1199_m_000002_0, Status : FAILED
>>>> Task attempt_201301230932_1199_m_000002_0 failed to report status for
>>>> 600 seconds. Killing!
>>>> 13/02/04 15:58:43 INFO mapred.JobClient: Task Id :
>>>> attempt_201301230932_1199_m_000000_0, Status : FAILED
>>>>
>>>>
>>>>
>>>>
>>>>========================================================================
>>>>=========================
>>>> This is the code I was using:
>>>>
>>>> import com.google.common.base.Preconditions;
>>>> import com.google.common.collect.Maps;
>>>>
>>>> import org.apache.giraph.comm.ArrayListWritable;
>>>> import org.apache.giraph.graph.BasicVertex;
>>>> import org.apache.giraph.graph.BspUtils;
>>>> import org.apache.giraph.graph.GiraphJob;
>>>> import org.apache.giraph.graph.EdgeListVertex;
>>>> import org.apache.giraph.graph.VertexReader;
>>>> import org.apache.giraph.graph.VertexWriter;
>>>> import org.apache.giraph.lib.TextVertexInputFormat;
>>>> import org.apache.giraph.lib.TextVertexInputFormat.TextVertexReader;
>>>> import org.apache.giraph.lib.TextVertexOutputFormat;
>>>> import org.apache.giraph.lib.TextVertexOutputFormat.TextVertexWriter;
>>>> import org.apache.hadoop.conf.Configuration;
>>>> import org.apache.hadoop.fs.Path;
>>>> import org.apache.hadoop.io.DoubleWritable;
>>>> import org.apache.hadoop.io.FloatWritable;
>>>> import org.apache.hadoop.io.IntWritable;
>>>> import org.apache.hadoop.io.LongWritable;
>>>> import org.apache.hadoop.io.Text;
>>>> import org.apache.hadoop.mapreduce.InputSplit;
>>>> import org.apache.hadoop.mapreduce.RecordReader;
>>>> import org.apache.hadoop.mapreduce.RecordWriter;
>>>> import org.apache.hadoop.mapreduce.TaskAttemptContext;
>>>> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>>>> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>>>> import org.apache.hadoop.util.Tool;
>>>> import org.apache.hadoop.util.ToolRunner;
>>>> import org.apache.log4j.Logger;
>>>> import org.json.JSONArray;
>>>> import org.json.JSONException;
>>>>
>>>> import java.io.IOException;
>>>> import java.util.ArrayList;
>>>> import java.util.Iterator;
>>>> import java.util.Map;
>>>> import java.util.StringTokenizer;
>>>>
>>>> /**
>>>>  * Shows an example of a brute-force implementation of the Travelling
>>>> Salesman Problem
>>>>  */
>>>> public class SimpleGiraphSumEdgeWeights extends
>>>> EdgeListVertex<LongWritable, ArrayListWritable<DoubleWritable>,
>>>> FloatWritable, ArrayListWritable<Text>> implements Tool {
>>>>     /** Configuration */
>>>>     private Configuration conf;
>>>>     /** Class logger */
>>>>     private static final Logger LOG =
>>>>         Logger.getLogger(SimpleGiraphSumEdgeWeights.class);
>>>>     /** The shortest paths id */
>>>>     public static String SOURCE_ID =
>>>>"SimpleShortestPathsVertex.sourceId";
>>>>     /** Default shortest paths id */
>>>>     public static long SOURCE_ID_DEFAULT = 1;
>>>>
>>>>
>>>>     /**
>>>>      * Is this vertex the source id?
>>>>      *
>>>>      * @return True if the source id
>>>>      */
>>>>     private boolean isSource() {
>>>>         return (getVertexId().get() ==
>>>>             getContext().getConfiguration().getLong(SOURCE_ID,
>>>>
>>>>SOURCE_ID_DEFAULT));
>>>>     }
>>>>     public class Message extends ArrayListWritable<Text> {
>>>>     public Message() {
>>>>       super();
>>>>     }
>>>>
>>>> @Override
>>>> public void setClass() {
>>>> // TODO Auto-generated method stub
>>>> }
>>>>   }
>>>>     public class Valeur extends ArrayListWritable<DoubleWritable> {
>>>>     public Valeur() {
>>>>       super();
>>>>     }
>>>>
>>>>   @Override
>>>>   public void setClass() {
>>>>   // TODO Auto-generated method stub
>>>>
>>>>   }
>>>>   }
>>>>
>>>>     @Override
>>>>     public void compute(Iterator<ArrayListWritable<Text>> msgIterator)
>>>>{
>>>>     System.out.println("****     LAUNCHING COMPUTATION FOR VERTEX
>>>> "+this.getVertexId().get()+", SUPERSTEP "+this.getSuperstep()+"
>>>> ****");
>>>>     //We get the source ID, we will need it
>>>>     String sourceID = new
>>>> LongWritable(this.getContext().getConfiguration().getLong(SOURCE_ID,
>>>>                 SOURCE_ID_DEFAULT)).toString();
>>>>     //We get the total number of verticles, and the current superstep
>>>> number, we will need it too
>>>>         int J=1;
>>>>
>>>>         voteToHalt();
>>>>     }
>>>>
>>>>     /**
>>>>      * VertexInputFormat that supports {@link
>>>>SimpleGiraphSumEdgeWeights}
>>>>      */
>>>>     public static class SimpleShortestPathsVertexInputFormat extends
>>>> TextVertexInputFormat<LongWritable, ArrayListWritable<DoubleWritable>,
>>>>                                   FloatWritable,
>>>>                                   DoubleWritable> {
>>>>         @Override
>>>>         public VertexReader<LongWritable,
>>>> ArrayListWritable<DoubleWritable>, FloatWritable, DoubleWritable>
>>>>                 createVertexReader(InputSplit split,
>>>>                                    TaskAttemptContext context)
>>>>                                    throws IOException {
>>>>             return new SimpleShortestPathsVertexReader(
>>>>                 textInputFormat.createRecordReader(split, context));
>>>>         }
>>>>     }
>>>>
>>>>     /**
>>>>      * VertexReader that supports {@link SimpleGiraphSumEdgeWeights}.
>>>>In
>>>> this
>>>>      * case, the edge values are not used.  The files should be in the
>>>>      * following JSON format:
>>>>      * JSONArray(<vertex id>, <vertex value>,
>>>>      *           JSONArray(JSONArray(<dest vertex id>, <edge value>),
>>>> ...))
>>>>      * Here is an example with vertex id 1, vertex value 4.3, and two
>>>> edges.
>>>>      * First edge has a destination vertex 2, edge value 2.1.
>>>>      * Second edge has a destination vertex 3, edge value 0.7.
>>>>      * [1,4.3,[[2,2.1],[3,0.7]]]
>>>>      */
>>>>     public static class SimpleShortestPathsVertexReader extends
>>>>             TextVertexReader<LongWritable,
>>>>             ArrayListWritable<DoubleWritable>, FloatWritable,
>>>> DoubleWritable> {
>>>>
>>>>         public SimpleShortestPathsVertexReader(
>>>>                 RecordReader<LongWritable, Text> lineRecordReader)
{
>>>>             super(lineRecordReader);
>>>>         }
>>>>
>>>>         public class Valeur extends ArrayListWritable<DoubleWritable>
{
>>>>           public Valeur() {
>>>>             super();
>>>>           }
>>>>
>>>>       @Override
>>>>       public void setClass() {
>>>>       // TODO Auto-generated method stub
>>>>
>>>>       }
>>>>         }
>>>>
>>>>         @Override
>>>>         public BasicVertex<LongWritable,
>>>> ArrayListWritable<DoubleWritable>, FloatWritable,
>>>>                            DoubleWritable> getCurrentVertex()
>>>>             throws IOException, InterruptedException {
>>>>           BasicVertex<LongWritable, ArrayListWritable<DoubleWritable>,
>>>> FloatWritable,
>>>>               DoubleWritable> vertex = BspUtils.<LongWritable,
>>>> ArrayListWritable<DoubleWritable>, FloatWritable,
>>>>
>>>> DoubleWritable>createVertex(getContext().getConfiguration());
>>>>
>>>>             Text line = getRecordReader().getCurrentValue();
>>>>             try {
>>>>                 JSONArray jsonVertex = new JSONArray(line.toString());
>>>>                 LongWritable vertexId = new
>>>> LongWritable(jsonVertex.getLong(0));
>>>>                Valeur vertexValue = new Valeur();
>>>>                vertexValue.add(new
>>>> DoubleWritable(jsonVertex.getDouble(1)));
>>>>                 Map<LongWritable, FloatWritable> edges =
>>>> Maps.newHashMap();
>>>>                 JSONArray jsonEdgeArray = jsonVertex.getJSONArray(2);
>>>>                 for (int i = 0; i < jsonEdgeArray.length(); ++i) {
>>>>                     JSONArray jsonEdge = jsonEdgeArray.getJSONArray(i);
>>>>                     edges.put(new LongWritable(jsonEdge.getLong(0)),
>>>>                             new FloatWritable((float)
>>>> jsonEdge.getDouble(1)));
>>>>                 }
>>>>                 vertex.initialize(vertexId, vertexValue, edges, null);
>>>>             } catch (JSONException e) {
>>>>                 throw new IllegalArgumentException(
>>>>                     "next: Couldn't get vertex from line " +
>>>> line.toString(), e);
>>>>             }
>>>>             return vertex;
>>>>         }
>>>>
>>>>         @Override
>>>>         public boolean nextVertex() throws IOException,
>>>> InterruptedException {
>>>>             return getRecordReader().nextKeyValue();
>>>>         }
>>>>     }
>>>>
>>>>     /**
>>>>      * VertexOutputFormat that supports {@link
>>>>SimpleGiraphSumEdgeWeights}
>>>>      */
>>>>     public static class SimpleShortestPathsVertexOutputFormat extends
>>>>             TextVertexOutputFormat<LongWritable,
>>>> ArrayListWritable<DoubleWritable>,
>>>>             FloatWritable> {
>>>>
>>>>         @Override
>>>>         public VertexWriter<LongWritable,
>>>> ArrayListWritable<DoubleWritable>, FloatWritable>
>>>>                 createVertexWriter(TaskAttemptContext context)
>>>>                 throws IOException, InterruptedException {
>>>>             RecordWriter<Text, Text> recordWriter =
>>>>                 textOutputFormat.getRecordWriter(context);
>>>>             return new SimpleShortestPathsVertexWriter(recordWriter);
>>>>         }
>>>>     }
>>>>
>>>>     /**
>>>>      * VertexWriter that supports {@link SimpleGiraphSumEdgeWeights}
>>>>      */
>>>>     public static class SimpleShortestPathsVertexWriter extends
>>>>             TextVertexWriter<LongWritable,
>>>> ArrayListWritable<DoubleWritable>, FloatWritable> {
>>>>         public SimpleShortestPathsVertexWriter(
>>>>                 RecordWriter<Text, Text> lineRecordWriter) {
>>>>             super(lineRecordWriter);
>>>>         }
>>>>
>>>>         @Override
>>>>         public void writeVertex(BasicVertex<LongWritable,
>>>> ArrayListWritable<DoubleWritable>,
>>>>                                 FloatWritable, ?> vertex)
>>>>                 throws IOException, InterruptedException {
>>>>         String sourceID = new
>>>> LongWritable(vertex.getContext().getConfiguration().getLong(SOURCE_ID,
>>>>                     SOURCE_ID_DEFAULT)).toString();
>>>>         JSONArray jsonVertex = new JSONArray();
>>>>             try {
>>>>                 jsonVertex.put(vertex.getVertexId().get());
>>>>                 jsonVertex.put(vertex.getVertexValue().toString());
>>>>                 JSONArray jsonEdgeArray = new JSONArray();
>>>>                 for (LongWritable targetVertexId : vertex) {
>>>>                     JSONArray jsonEdge = new JSONArray();
>>>>                     jsonEdge.put(targetVertexId.get());
>>>>
>>>> jsonEdge.put(vertex.getEdgeValue(targetVertexId).get());
>>>>                     jsonEdgeArray.put(jsonEdge);
>>>>                 }
>>>>                 jsonVertex.put(jsonEdgeArray);
>>>>             } catch (JSONException e) {
>>>>                 throw new IllegalArgumentException(
>>>>                     "writeVertex: Couldn't write vertex " + vertex);
>>>>             }
>>>>             getRecordWriter().write(new Text(jsonVertex.toString()),
>>>> null);
>>>>         }
>>>>     }
>>>>
>>>>     @Override
>>>>     public Configuration getConf() {
>>>>         return conf;
>>>>     }
>>>>
>>>>     @Override
>>>>     public void setConf(Configuration conf) {
>>>>         this.conf = conf;
>>>>     }
>>>>
>>>>     @Override
>>>>     public int run(String[] argArray) throws Exception {
>>>>         Preconditions.checkArgument(argArray.length == 4,
>>>>             "run: Must have 4 arguments <input path> <output path>
" +
>>>>             "<source vertex id> <# of workers>");
>>>>
>>>>         GiraphJob job = new GiraphJob(getConf(), getClass().getName());
>>>>         job.setVertexClass(getClass());
>>>>         job.setVertexInputFormatClass(
>>>>             SimpleShortestPathsVertexInputFormat.class);
>>>>         job.setVertexOutputFormatClass(
>>>>             SimpleShortestPathsVertexOutputFormat.class);
>>>>         FileInputFormat.addInputPath(job, new Path(argArray[0]));
>>>>         FileOutputFormat.setOutputPath(job, new Path(argArray[1]));
>>>>
>>>> job.getConfiguration().setLong(SimpleGiraphSumEdgeWeights.SOURCE_ID,
>>>>                                        Long.parseLong(argArray[2]));
>>>>         job.setWorkerConfiguration(Integer.parseInt(argArray[3]),
>>>>                                    Integer.parseInt(argArray[3]),
>>>>                                    100.0f);
>>>>
>>>>         return job.run(true) ? 0 : -1;
>>>>     }
>>>>
>>>>     public static void main(String[] args) throws Exception {
>>>>         System.exit(ToolRunner.run(new SimpleGiraphSumEdgeWeights(),
>>>> args));
>>>>     }
>>>> }
>>>>
>>>> On Fri, Feb 1, 2013 at 5:37 PM, Eli Reisman <apache.mailbox@gmail.com>
>>>> wrote:
>>>> > Your best bet is to look over the two code components that users most
>>>> > often
>>>> > have to tweak or implement to write application code. That is, the
>>>> > Vertex
>>>> > implementations in examples/ and benchmark/ and the IO formats and
>>>> > related
>>>> > goodies like RecordReaders etc. that are mostly in the io/ dir. You
>>>> > might
>>>> > also take a look at the test suite for some quick ideas of how some
>>>>of
>>>> > the
>>>> > moving parts fit together.
>>>> >
>>>> > If you have real work to do with Giraph, you're going to need to get
>>>> > used to
>>>> > 0.2 and its API. The old API is both limited in what kind of data it
>>>> > will
>>>> > process, and not compatible into the future. The API we have now,
>>>>while
>>>> > evolving, is much much closer to being "final" than anything in 0.1
>>>>And
>>>> > regardless, we now have (in hindsight) the sure knowledge that none
>>>>of
>>>> > the
>>>> > code you write for 0.1 will be portable into the future.
>>>> >
>>>> > I am first in line to be sorry about the state of the docs. There are
>>>> > efforts underway now to fix this.  We all owe the users a collective
>>>> > apology
>>>> > for this. In lieu of proper apologies, feel free to ask any and all
>>>> > questions, no matter how dumb, they can't be as dumb as mine! The
>>>> > codebase
>>>> > is under heavy development and has a lot of confusingly-named moving
>>>> > parts
>>>> > so first get used to the plumbing an app writer has to know to
>>>>function,
>>>> > get
>>>> > some apps up and running, then dig into the framework code and it
>>>>will
>>>> > make
>>>> > more sense.
>>>> >
>>>> > One string to pull on to begin to look inside the framework is
>>>> > bin/giraph ->
>>>> > org.apache.giraph.GiraphRunner (hands job to Hadoop) -> ... ->
>>>> > o.a.g.graph.GraphMapper (is a mapper instance on a Hadoop cluster,
>>>> > started
>>>> > according to the Job sumbitted to Hadoop, but running our BSP code
>>>> > instead)
>>>> > -> o.a.g.graph.GraphTaskManager -> lots of places from there...
>>>> >
>>>> > The overarching BSP activity management for a single job run is
>>>> > basically
>>>> > all stemming out of GraphTaskManager now. You can look at setup() and
>>>> > execute() and get a decent idea of the major events in a job run, and
>>>> > where
>>>> > to look to get a better peek under the hood at any given task or
>>>>event.
>>>> > Good
>>>> > luck!
>>>> >
>>>> >
>>>> > On Fri, Feb 1, 2013 at 4:59 PM, Gustavo Enrique Salazar Torres
>>>> > <gsalazar@ime.usp.br> wrote:
>>>> >>
>>>> >> Hi Ryan:
>>>> >>
>>>> >> It's the simplest thing:
>>>> >> 1. Define your type of parameters for a type of Vertex (for example
>>>> >> EdgeListVertex)
>>>> >> 2. Implement compute method.
>>>> >>
>>>> >> From what I saw out there in the M/R world, Giraph provides the
>>>> >> simplest
>>>> >> way to work with graphs.
>>>> >>
>>>> >> Take a look at
>>>> >>
>>>> >>
>>>>https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Exampl
>>>>e
>>>> >> and use release 0.1
>>>> >> (http://www.apache.org/dyn/closer.cgi/incubator/giraph/)
>>>> >> because 0.2-SNAPSHOT is under heavy work.
>>>> >>
>>>> >> Hope this helps you.
>>>> >>
>>>> >> Gustavo
>>>> >>
>>>> >> On Fri, Feb 1, 2013 at 9:17 PM, Ryan Compton
>>>><compton.ryan@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> I am having trouble understand what all the classes do and the
>>>> >>> documentation looks like it might be out of date. I searched
around
>>>> >>> and found this: https://github.com/edaboussi/Giraph but it won't
>>>> >>> compile with 0.2, any suggestions?
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >
>

Mime
View raw message