giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamer Yousef <TYou...@boardreader.com>
Subject How to Write to HDFS?
Date Thu, 02 Oct 2014 20:35:28 GMT
Hello All!
I’m learning Giraph and trying few things, but I fail to write out output to hdfs. I created
my own .java file, and I placed it in the folder  ${GIRAPH_HOME}//giraph-examples/src/main/java/org/apache/giraph/examples/
then I ran the mvn compile to get a jar file that includes my class. The function is doing
nothing other than trying to:

1-       Write using the stdout

2-       Write using log4j

The program runs and it creates an output directory in hdfs as I specify in the command below,
but the output file does not reflect what the program should write out.
Here is the output I get in the output file in HDFS (the vertices  I have are very similar):

6              6
5              5
13           13
12           12
8              8
7              7
2              2
15           15
9              9
16           16
10           10
1              1
3              3
14           14
11           11
4              4

Even if I completely comment out the code in the compute class, I still get the output above
(with keeping the voteToHalt method).
I execute the code using the command:

hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.HelloWorld -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat
-vip /in/graph2.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /out5
-w 1

I’m working with Hadoop 1.2.1 and the latest Giraph from the trunk.

and here is my full class:

package org.apache.giraph.examples;
import org.apache.giraph.GiraphRunner;
import org.apache.hadoop.util.ToolRunner;
import org.apache.giraph.graph.BasicComputation;
import org.apache.giraph.conf.LongConfOption;
import org.apache.giraph.edge.Edge;
import org.apache.giraph.graph.Vertex;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.FloatWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.log4j.Logger;

import java.io.IOException;

@Algorithm(
    name = "Hellow",
    description = "test class"
)
  public class HelloWorld extends
                        BasicComputation<IntWritable, IntWritable,
                        NullWritable, NullWritable> {
                        @Override
                        public void compute(Vertex<IntWritable,
                                                IntWritable, NullWritable> vertex,
                                                Iterable<NullWritable> messages) {

                                    System.out.println("Hello world from print ln");
                                    LOG.info("Hello world from log info");

                                    vertex.voteToHalt();
                        }
                        public static void main(String[] args) throws Exception
                        {
                                    //log4j.logger.org.apache.hadoop = DEBUG;
                                    System.exit(ToolRunner.run(new GiraphRunner(), args));
                        }
                        /** Class logger */
                        private static final Logger LOG =
                                                Logger.getLogger(SimpleShortestPathsComputation.class);

            }

Any ideas?

Thanks!
Mime
View raw message