giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charith Wickramarachchi <charith.dhanus...@gmail.com>
Subject Re: How to Write to HDFS?
Date Thu, 02 Oct 2014 20:52:30 GMT
Hi Tamer,

The reason you see this behavior is IntIntNullTextInputFormat sets the
value of the vertex as same as the vertex id when creating a vertex. Since
you do not change the value vertex id will be written to the output as the
vertex value.

See the class org.apache.giraph.io.formats.IntIntNullTextInputFormat.IntIntNullVertexReader
and you will understand.

Hope this helps.

Thanks,
Charith


On Thu, Oct 2, 2014 at 1:35 PM, Tamer Yousef <TYousef@boardreader.com>
wrote:

>    Hello All!
>
> I’m learning Giraph and trying few things, but I fail to write out output
> to hdfs. I created my own .java file, and I placed it in the folder  ${GIRAPH_HOME}//giraph-examples/src/main/java/org/apache/giraph/examples/
> then I ran the mvn compile to get a jar file that includes my class. The
> function is doing nothing other than trying to:
>
> 1-       Write using the stdout
>
> 2-       Write using log4j
>
>
>
> The program runs and it creates an output directory in hdfs as I specify
> in the command below, but the output file does not reflect what the program
> should write out.
>
> Here is the output I get in the output file in HDFS (the vertices  I have
> are very similar):
>
>
>
> 6              6
>
> 5              5
>
> 13           13
>
> 12           12
>
> 8              8
>
> 7              7
>
> 2              2
>
> 15           15
>
> 9              9
>
> 16           16
>
> 10           10
>
> 1              1
>
> 3              3
>
> 14           14
>
> 11           11
>
> 4              4
>
>
>
> Even if I completely comment out the code in the compute class, I still
> get the output above (with keeping the voteToHalt method).
>
> I execute the code using the command:
>
>
>
> hadoop jar
> $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner org.apache.giraph.examples.HelloWorld -vif
> org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /in/graph2.txt
> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /out5 -w 1
>
>
>
> *I’m working with Hadoop 1.2.1 and the latest Giraph from the trunk.*
>
>
>
> *and here is my full class:*
>
>
>
> *package* org.apache.giraph.examples;
>
> *import* org.apache.giraph.GiraphRunner;
>
> *import* org.apache.hadoop.util.ToolRunner;
>
> *import* org.apache.giraph.graph.BasicComputation;
>
> *import* org.apache.giraph.conf.LongConfOption;
>
> *import* org.apache.giraph.edge.Edge;
>
> *import* org.apache.giraph.graph.Vertex;
>
> *import* org.apache.hadoop.io.IntWritable;
>
> *import* org.apache.hadoop.io.NullWritable;
>
> *import* org.apache.hadoop.io.DoubleWritable;
>
> *import* org.apache.hadoop.io.FloatWritable;
>
> *import* org.apache.hadoop.io.LongWritable;
>
> *import* org.apache.log4j.Logger;
>
>
>
> *import* java.io.IOException;
>
>
>
> @Algorithm(
>
>     name = "Hellow",
>
>     description = "test class"
>
> )
>
>   *public* *class* HelloWorld *extends*
>
>                         BasicComputation<IntWritable, IntWritable,
>
>                         NullWritable, NullWritable> {
>
>                         @Override
>
>                         *public* *void* compute(Vertex<IntWritable,
>
>                                                 IntWritable, NullWritable>
> vertex,
>
>                                                 Iterable<NullWritable>
> messages) {
>
>
>
>                                     System.out.println("Hello world from
> print ln");
>
>                                     LOG.info("Hello world from log info");
>
>
>
>                                     vertex.voteToHalt();
>
>                         }
>
>                         *public* *static* *void* main(String[] args)
> *throws* Exception
>
>                         {
>
>                                     //log4j.logger.org.apache.hadoop =
> DEBUG;
>
>                                     System.exit(ToolRunner.run(*new*
> GiraphRunner(), args));
>
>                         }
>
>                         /** Class logger */
>
>                         *private* *static* *final* Logger LOG =
>
>
> Logger.getLogger(SimpleShortestPathsComputation.*class*);
>
>
>
>             }
>
>
>
> *Any ideas?*
>
>
>
> *Thanks!*
>



-- 
Charith Dhanushka Wickramaarachchi

Tel  +1 213 447 4253
Web  http://apache.org/~charith <http://www-scf.usc.edu/~cwickram/>
<http://charith.wickramaarachchi.org/>
Blog  http://charith.wickramaarachchi.org/
<http://charithwiki.blogspot.com/>
Twitter  @charithwiki <https://twitter.com/charithwiki>

This communication may contain privileged or other confidential information
and is intended exclusively for the addressee/s. If you are not the
intended recipient/s, or believe that you may have
received this communication in error, please reply to the sender indicating
that fact and delete the copy you received and in addition, you should not
print, copy, retransmit, disseminate, or otherwise use the information
contained in this communication. Internet communications cannot be
guaranteed to be timely, secure, error or virus-free. The sender does not
accept liability for any errors or omissions

Mime
View raw message