giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Panagiotis Eustratiadis <ep.pan....@gmail.com>
Subject Re: giraph-netbeans -simple question
Date Fri, 22 May 2015 16:19:23 GMT
Hello Anna, you happened to answer while I was answering to the previous
e-mail. I'm glad you managed to set up a maven project with Junit.

To answer to you other questions, the MasterCompute class need not be
overriden, unless you need it. As you may have read in the documentation, a
MasterCompute computation is executed before each superstep. By default
every configuration has the DefaultMasterCompute, which does nothing. If
you need to take actions between supersteps (i.e use of aggregators) you
need to extend it. After you do that, you usually need to @Override just 2
methods: compute and initialize (initialize is for registering aggregators
and stuff).

Since you are a new Giraph user i would encourage you to invest some time
in investigating the website, and reading the documentation. There are some
non-obvious useful resources, and the website is much improved compared to
when I did my dissertation.

Also, you may not necessarily need both EdgeInputFormat and
VertexInputFormat, I just layed out all the possible options for you in my
example. Use the ones you need.

By the way avoid System.out.println when running Giraph tasks. Prefer to
use a logger.

Regards,
Panagiotis Eustratiadis.


On 22 May 2015 at 19:02, Anna Xenaki <axenaki@mail.ntua.gr> wrote:

>  Finaly I got a maven project up and running Depedencies and all. I added
> a test class this is what I get
> public class KatzReduceTest {
>
>     public KatzReduceTest() {
>     }
>
>     @BeforeClass
>     public static void setUpClass() {
>     }
>
>     @AfterClass
>     public static void tearDownClass() {
>     }
>
>     @Before
>     public void setUp() {
>     }
>
>     @After
>     public void tearDown() {
>     }
>
>     /**
>      * Test of compute method, of class KatzReduce.
>      */
>     @Test
>     public void testCompute() throws Exception {
>         System.out.println("compute");
>         String[] tinyGraph =
> {"1#2,3#54,78","2#1,3,4#78,63","3#1,2,4#77,125","4#2,3#77,63" };
>         // This is where you configure your job
>         GiraphConfiguration conf = new GiraphConfiguration();
>         conf.setComputationClass(KatzReduce.class);
>   conf.setMasterComputeClass();
>             conf.setEdgeInputFormatClass();
>     conf.setVertexInputFormatClass();
>
>         conf.setVertexInputFormatClass(FbGraphInputFormat.class);
>         conf.setVertexOutputFormatClass(FbGraphOutputFormat.class);
>         // Run and print results
>         Iterable<String> results = InternalVertexRunner.run(conf,
> tinyGraph, tinyGraph);
>         for (String result : results) {
>
>         }
>     }
> would you kindly provide me with an example  as to what to add in
> MasterComputeClass (difference with simple computationclass) and
> EdgeInputFormat , VertexInputFormat (should those two be totally seperate?
> and If so in Edge input format how do I add the edge (Source-Target
> ValueId) if I dont have a vertex created when I read the input
>
> Thanks in advance, Anna
>
> On 22/5/2015 5:53 μμ, Anna Xenaki wrote:
>
> Hi, I dont have a maven project, I have never created one before so I dont
> know the process. I have a few questions:
> 1. If I compile it from terminal how do I do that? Do I have to create a
> FbGraphRunner class? make the whole project a jar? and then run " $HADOOP_HOME/bin/hadoop
> jar my.jar my.project.folder.myFbGraphRunner my.project.folder.KatzReduce
> -vif my.project.folder.FbGraphInputFormat -vip /input/fb0.txt -vof
> my.project.folder.FbGraphOutputFormat -op /output/katz1.txt -w 1 -ca
> giraph.SplitMasterWorker=false" ?
> 2. Do I need to have a class that extends MasterCompute class?
> 3.When I create a Maven project for giraph what are my dependencies? do I
> only put the one you mentioned below? Do I have any libraries (like the
> ones I added in my java project)?
> 4.In your suggestion you mentioned   EdgeInputFormatClass and
> VertexInputFormatClass in order to import a Graph in Giraph do I need to
> have both?To  Import edges and vertexes separately I currently have only
> one InputFormat and add edges as a List with createEdgeFactory
>
> Best Regards, Anna
>
> On 22/5/2015 4:16 μμ, Panagiotis Eustratiadis wrote:
>
>  Hello Anna,
>
>  I use two ways to run a Giraph job for testing. The one is to setup a
> single node cluster (there are many tutorials on that) and run your job via
> the command line.
>
>  The other way (through NetBeans) is Junit testing. Add those lines to
> your pom:
>
> <dependency>
>     <groupId>junit</groupId>
>     <artifactId>junit</artifactId>
>     <version>4.11</version>
> </dependency>
>
>  Then create a test package in your netbeans project. Normally there are
> two directories under your_root/src called "main" and "test". If there
> isn't one named "test", you should create and structure it normally, like
> the other one (add "java" and "resources" directories, packages, etc.).
> After that you will be able to see it through your IDE as well, and manage
> it like the main package. Finally, you can write Junit tests. Create a
> class called "Tests.java" or whatever else you want, and write methods in
> it using the @Test annotation. For example:
>
> @Test
> public void yourTestFunction() throws Exception {
>
>     String[] tinyGraph = {
>         "1 2", "1 3",
>         "2 1", "2 4",
>         "3 1", "3 4",
>         "4 2", "4 3", "4 5", "4 6",
>         "5 4", "5 7",
>         "6 4", "6 7",
>         "7 5", "7 6"
>     };
>
>      // This is where you configure your job
>      GiraphConfiguration conf = new GiraphConfiguration();
>     conf.setComputationClass();
>     conf.setMasterComputeClass();
>     conf.setEdgeInputFormatClass();
>     conf.setVertexInputFormatClass();
>     conf.setVertexOutputFormatClass();
>
>      // Run and print results
>      Iterable<String> results = InternalVertexRunner.run(conf, tinyGraph,
> tinyGraph);
>     for (String result : results) {
>         LOGGER.debug(result);
>     }
>
> }
>
>  Note that the tests will run every time you compile it, unless you
> specify -DskipTests in the mvn command.
>  Regards,
>  Panagiotis Eustratiadis.
>
>
> On 22 May 2015 at 15:43, Anna Xenaki <axenaki@mail.ntua.gr> wrote:
>
>> Hi, sorry for the inconvenience I found your mail via giraph mailing
>> list. Im new in Giraph.
>> I want to run a giraph job via netbeans. I have imported all libs in
>> project , and installed hadoop 2.5.2 Giraph 1.1.0 in ubuntu 14.04.
>> I created Java classes FbGraphInputFormat, FbGraphMessage,
>> FbGraphOutputFormat,FbGraphState, KatzReduce
>>
>> public class KatzReduce extends BasicComputation<Text, FbGraphState,
>> DoubleWritable, FbGraphMessage> {
>> public class FbGraphState implements Writable {
>> public class FbGraphOutputFormat  extends
>> TextVertexOutputFormat<Text,FbGraphState,DoubleWritable>{
>> public class FbGraphInputFormat extends TextVertexInputFormat<Text,
>> FbGraphState, DoubleWritable> {
>> class FbGraphMessage implements Writable{
>>
>>
>> MY QUESTION is simple I want to run my java project from netbeans I dont
>> know how . I dont have a main class and do not know to start a giraph job
>> that does exactly what KatzReduce.java says.
>>
>> I added in the run of the project the arguments infile_location_path
>> outfile_location_path
>>
>> Do I create a main method saying what? I cannot find any documentation on
>> deploying giraph projects from scratch so any help would be of great
>> assistance.
>>
>> Thanks in advance, Anna
>>
>
>
>
>

Mime
View raw message