Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7908817AF5 for ; Mon, 4 May 2015 17:18:32 +0000 (UTC) Received: (qmail 49211 invoked by uid 500); 4 May 2015 17:18:32 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 49159 invoked by uid 500); 4 May 2015 17:18:32 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 49149 invoked by uid 99); 4 May 2015 17:18:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 May 2015 17:18:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which is an MX secondary for user@giraph.apache.org) Received: from [54.76.25.247] (HELO mx1-eu-west.apache.org) (54.76.25.247) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 May 2015 17:18:06 +0000 Received: from mail-wg0-f49.google.com (mail-wg0-f49.google.com [74.125.82.49]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 05DAA20317 for ; Mon, 4 May 2015 17:18:04 +0000 (UTC) Received: by wgso17 with SMTP id o17so157279482wgs.1 for ; Mon, 04 May 2015 10:16:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=UrZAGrl1v26g/L39NIOLQWIsCTf4War7PjcMu3K5WC4=; b=MNQGps8bvHPNzdkLAT0k2wLbNjUboJNbykHfqUKaJvt5ih7/dGf44a6EwSHTavXrTv WuB6Wn35oDPgLxARP+WRH5qBuzHcgWPEdlo30mz+Gu1keDaQQWxBlwbH+CIYRpWy+MxG B+kw48pIdHuvjAW+Tzyo3M4GxdqB0xfuhfVHb7DZ3TjyJv8vlq4gqvtYS2LCPYNetHyh fMNd/qlzcuKlaJtaE0lSZNju20A59tS1JWRgdDOjArThzT0WkfwOSmgSmAB3SFmG65u5 PbaL1JvhkSkHA3NwqxjKisVIvhXCds/1t6Tcdmrrcg5veefP7X/nfBZUh/QbeR9BxQei bzZQ== X-Gm-Message-State: ALoCoQm+pEKR/AvkEfxAFXQFHPB5rb/CG+lYuvboTdEO8IhoEiaCtL8bb60d6JkjsKDFKoDK04Ra X-Received: by 10.194.6.37 with SMTP id x5mr44593370wjx.73.1430759788056; Mon, 04 May 2015 10:16:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.27.201.2 with HTTP; Mon, 4 May 2015 10:16:07 -0700 (PDT) In-Reply-To: References: From: Steven Harenberg Date: Mon, 4 May 2015 13:16:07 -0400 Message-ID: Subject: Re: Input format problems running Giraph 1.1.0 on Twitter dataset To: user Content-Type: multipart/alternative; boundary=047d7b5d9acded2993051544b897 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b5d9acded2993051544b897 Content-Type: text/plain; charset=UTF-8 My understanding is that a vertex with only incoming edges will not be active until it receives a message, which is why you don't see all of the vertices initially. The easiest way to test this is to write a script that parses your input and creates a new data file where every vertex is specified on a line of its own. Even if it has no outgoing neighbors, just leave the neighbor empty. Or, first just check if you have 40383589-40103281=280308 vertices with only incoming edges. Young provided another solution for fixing the initialization problem, and it looks like in the code that wasn't specified this code to still have the problem. Either transform the input (seems like the easiest thing to do), or try the fix Young said. I would bet either of those would fix the issue. Young may have better ideas since he seems more experienced with Giraph than I am. --Steve On Sat, May 2, 2015 at 2:19 PM, Kenrick Fernandes wrote: > Thank you both for your responses. > > Steve, I faced the same problem when I created the Long input format > files. > I tried running the code linked by Young above, using the > *SimplePageRankInputFormat.java* > as well as the *SimplePageRankVertex.java* in the repo. > > For the Twitter dataset, I added some *MasterCompute* code to log the > number of vertices > that existed at each superstep. The results, however, look pretty similar > to the previous iteration: > > Current step is 1 - 40103281 existed in the previous superstep 0Current step is 2 - 40103281 existed in the previous superstep 1 > > Current step is 3 - 40383589 existed in the previous superstep 2 > > Current step is 31 - 40383589 existed in the previous superstep 30 > > It seems that a subset of vertices still only become active after the > first superstep, > despite all vertices being initialized in superstep 0. I cant think of a > reason why > - thoughts ? > > Thanks, > Kenrick > > > > On Wed, Apr 29, 2015 at 2:33 PM, Young Han wrote: > >> For the initialization issue, you can define a (nested) class that >> extends DefaultVertexValueFactory (from org.apache.giraph.factories) and >> add >> "-Dgiraph.vertexValueFactoryClass=org.apache.giraph.examples.AlgClass\$AlgVertexValueFactory" >> after "org.apache.giraph.GiraphRunner" in your hadoop jar command. >> >> Also, the reason those input formats don't work is because PageRank is >> using LongWritable for vertex id and DoubleWritable for vertex value. As >> Roman pointed out, you have to have an input class that matches it (even if >> the input dataset has no "double" vertex values). An example (for Giraph >> 1.0.0) can be found here: >> https://github.com/xvz/graph-processing/blob/master/giraph-1.0.0/giraph-examples/src/main/java/org/apache/giraph/examples/SimplePageRankInputFormat.java >> and an example command that uses it here: >> https://github.com/xvz/graph-processing/blob/master/benchmark/giraph/pagerank.sh#L50 >> >> Young >> >> On Wed, Apr 29, 2015 at 11:24 AM, Steven Harenberg >> wrote: >> >>> Hey Kenrick, >>> >>> First, your commands above are wrong since you are specifying adjacency >>> list format with the -vif argument and since I believe *LongLongNullTextInputFormat >>> *refers to adjacency list format. However, even with the right commands >>> there will be issues and more things you need to do. >>> >>> I did get it the edgelist input format to work by creating a >>> LongNullTextEdgeInputFormat.java file just like the >>> giraph-core/src/main/java/org/apache/giraph/io/formats/IntNullTextEdgeInputFormat.java >>> file, but with longs instead of ints (this also required creating a >>> LongPair class). >>> >>> However, I would advise against using an edgelist input format in Giraph >>> as there are major underlying issues that I never figured out how to >>> resolve. Namely, for an edgelist format, Giraph only considers a vertex >>> active in the first superstep if it has an outgoing edge. This means that >>> vertices with only incoming edges won't be initialized with correct values >>> during things like PageRank, SSSP, or WCC and hence will output incorrect >>> results. (You can see my previous thread here: >>> http://mail-archives.apache.org/mod_mbox/giraph-user/201502.mbox/%3CCAHv2Baw7zFJ-s7dtNMv5dkNxz_zE436krE%2B6G4r3tp-HVgjW2g%40mail.gmail.com%3E >>> ) >>> >>> The above issue can be avoided with adjacency list format by specifying >>> the vertex with no neighbors. For example, if vertex v has only incoming >>> edges, then you make sure there is a line with just v and no neighbors >>> listed ( >>> http://mail-archives.apache.org/mod_mbox/giraph-user/201408.mbox/%3C1409255770206.93691@uiowa.edu%3E >>> ). >>> >>> If you figure out how to resolve the edgelist input issue please let me >>> know. >>> >>> Regards, >>> Steve >>> >>> >>> On Sat, Apr 25, 2015 at 9:54 PM, Kenrick Fernandes < >>> kenrick.f15@gmail.com> wrote: >>> >>>> Hi Roman, >>>> >>>> Thanks for the quick response. There is no vertex data in this >>>> dataset though, and the vertex IDs posted above would fit in a >>>> Long. Would you advise changing the PageRankComputation >>>> formats, or working on a new input format ? >>>> >>>> Thanks, >>>> Kenrick >>>> >>>> On Sat, Apr 25, 2015 at 7:40 PM, Roman Shaposhnik >>> > wrote: >>>> >>>>> One of the slightly annoying things in Giraph is that you have >>>>> to manually match your input format to your computation. In >>>>> your case, PageRankComputation requires LongWritable for >>>>> vertex ID and DoubleWritable for vertex Data. You may need >>>>> to hack one of the existing formats slightly. >>>>> >>>>> >>>>> Thanks, >>>>> Roman. >>>>> >>>>> On Sat, Apr 25, 2015 at 2:58 PM, Kenrick Fernandes >>>>> wrote: >>>>> > Hello, >>>>> > >>>>> > Im trying to get Giraph to read the Twitter dataset as input for the >>>>> > SimplePageRankComputation program. The dataset format looks like >>>>> this: >>>>> > 61578010 61147436 >>>>> > 61578037 61147436 >>>>> > 61578040 61147436 >>>>> > (vertex id's, with pairs representing edges) >>>>> > >>>>> > When I run the command with >>>>> > -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat, I get >>>>> this >>>>> > error : >>>>> > java.lang.IllegalArgumentException: checkClassTypes: vertex index >>>>> types not >>>>> > assignable, computation - class org.apache.hadoop.io.LongWritable, >>>>> > VertexInputFormat - class org.apache.hadoop.io.IntWritable >>>>> > >>>>> > So I tried running the command with >>>>> > -vif org.apache.giraph.io.formats.LongLongNullTextInputFormat and I >>>>> get a >>>>> > different one: >>>>> > java.lang.IllegalArgumentException: checkClassTypes: vertex value >>>>> types not >>>>> > assignable, computation - class org.apache.hadoop.io.DoubleWritable, >>>>> > VertexInputFormat - class org.apache.hadoop.io.LongWritable >>>>> > >>>>> > I dont understand why the types in the input show up as different >>>>> formats in >>>>> > each error. Also, as far as I could tell, there is no input format >>>>> for >>>>> > DoubleDouble. Is there a different way to get the graph into Giraph >>>>> without >>>>> > having to write custom input code ? Thoughts would be much >>>>> appreciated. >>>>> > >>>>> > ----- >>>>> > Reference Command: >>>>> > hadoop jar >>>>> giraph-examples-1.1.0-for-hadoop-1.1.2-jar-with-dependencies.jar >>>>> > org.apache.giraph.GiraphRunner >>>>> > org.apache.giraph.examples.PageRankComputation -vif >>>>> > org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip >>>>> > /user/kenrick/twitter/input -op /user/kenrick/twitter/output -w 30 >>>>> > ----- >>>>> > >>>>> > Thanks, >>>>> > Kenrick >>>>> >>>> >>>> >>> >> > --047d7b5d9acded2993051544b897 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My understanding is that a vertex with only= incoming edges will not be active until it receives a message, which is wh= y you don't see all of the vertices initially. The easiest way to test = this is to write a script that parses your input and creates a new data fil= e where every vertex is specified on a line of its own. Even if it has no o= utgoing neighbors, just leave the neighbor empty. Or, first just check if y= ou have 40383589-40103281=3D280308 vertices wi= th only incoming edges.

Young provided another solutio= n for fixing the initialization problem, and it looks like in the code that= wasn't specified this code to still have the problem.

Eit= her transform the input (seems like the easiest thing to do), or try the fi= x Young said. I would bet either of those would fix the issue. Young may ha= ve better ideas since he seems more experienced with Giraph than I am.

--Steve
<= br>
On Sat, May 2, 2015 at 2:19 PM, Kenrick Ferna= ndes <kenrick.f15@gmail.com> wrote:
Thank you both for your = responses.=C2=A0

Steve, I faced the same problem when I = created the Long input format files.=C2=A0
I tried running the co= de linked by Young above, using the SimplePageRankInputFormat.java
as well as the SimplePageRankVertex.java=C2=A0in the repo.= =C2=A0

For the Twitter dataset, I added some Ma= sterCompute=C2=A0code to log the number of vertices
that exis= ted at each superstep. The results, however, look pretty similar to the pre= vious iteration:

Curr=
ent step is 1 - 40103281 existed in the previous superstep 0
Current step is 2 - 40103281 existed in the previou=
s superstep 1
Current step is 3 - 40383589 existed in the pr=
evious superstep 2

Current step is 31 - 40383589 existed i=
n the previous superstep 30
It seems that a subset of vertices still only b=
ecome active after the first superstep,=C2=A0
despite all vertices being initializ=
ed in superstep 0. I cant think of a reason why
- thoughts ?

Thanks,
Kenrick



On Wed, Apr 29, 2015 at 2:33 PM, Young Han <<= a href=3D"mailto:young.han@uwaterloo.ca" target=3D"_blank">young.han@uwater= loo.ca> wrote:
For the initialization issue, you can define a= (nested) class that extends DefaultVertexValueFactory (from org.apache.gir= aph.factories) and add "-Dgiraph.vertexValueFactoryClass=3Dorg.apache.= giraph.examples.AlgClass\$AlgVertexValueFactory" after "org.apach= e.giraph.GiraphRunner" in your hadoop jar command.

A= lso, the reason those input formats don't work is because PageRank is u= sing LongWritable for vertex id and DoubleWritable for vertex value. As Rom= an pointed out, you have to have an input class that matches it (even if th= e input dataset has no "double" vertex values). An example (for G= iraph 1.0.0) can be found here: https://gi= thub.com/xvz/graph-processing/blob/master/giraph-1.0.0/giraph-examples/src/= main/java/org/apache/giraph/examples/SimplePageRankInputFormat.java and= an example command that uses it here: https://github.com/xvz/graph-processing/blob/master/benchmark/giraph/pa= gerank.sh#L50

Young

On We= d, Apr 29, 2015 at 11:24 AM, Steven Harenberg <sdharenb@ncsu.edu> wrote:
Hey Kenrick,

First, your comm= ands above are wrong since you are specifying adjacency list format with th= e -vif argument and since I believe LongLongNullTextInputFormat refe= rs to adjacency list format. However, even with the right commands there wi= ll be issues and more things you need to do.

I did get it the = edgelist input format to work by creating a LongNullTextEdgeInputFormat.jav= a file just like the giraph-core/src/main/java/org/apache/giraph/io/formats= /IntNullTextEdgeInputFormat.java file, but with longs instead of ints (this= also required creating a LongPair class).

However, I would ad= vise against using an edgelist input format in Giraph as there are major un= derlying issues that I never figured out how to resolve. Namely, for an edg= elist format, Giraph only considers a vertex active in the first superstep = if it has an outgoing edge. This means that vertices with only incoming edg= es won't be initialized with correct values during things like PageRank= , SSSP, or WCC and hence will output incorrect results. (You can see my pre= vious thread here: http://mail-archives.apache.org/mo= d_mbox/giraph-user/201502.mbox/%3CCAHv2Baw7zFJ-s7dtNMv5dkNxz_zE436krE%2B6G4= r3tp-HVgjW2g%40mail.gmail.com%3E)

The above issue can be a= voided with adjacency list format by specifying the vertex with no neighbor= s. For example, if vertex v has only incoming edges, then you make sure the= re is a line with just v and no neighbors listed (http://mail-archives.apache.org/mod_mbox/gir= aph-user/201408.mbox/%3C1409255770206.93691@uiowa.edu%3E).
If you figure out how to resolve the edgelist input issue please let me kn= ow.

Regards,
Steve
<= div>


On Sat, Apr 25, 2015 at 9:54 PM, Kenrick Fernandes <kenrick.f15@gmail.com> wrote:
Hi Roman,

Thank= s for the quick response. There is no vertex data in this
dataset= though, and the vertex IDs posted above would fit in a=C2=A0
Lon= g. Would you advise changing the PageRankComputation
formats, or = working on a new input format ?

Thanks,
= Kenrick

On Sat, Apr 25, 2015 at 7:40 PM, Roman Shaposhnik <roman@sh= aposhnik.org> wrote:
One of the slightly annoying things in Giraph is that you have=
to manually match your input format to your computation. In
your case, PageRankComputation requires LongWritable for
vertex ID and DoubleWritable for vertex Data. You may need
to hack one of the existing formats slightly.


Thanks,
Roman.

On Sat, Apr 25, 2015 at 2:58 PM, Kenrick Fernandes
<kenrick.f15@= gmail.com> wrote:
> Hello,
>
> Im trying to get Giraph to read the Twitter dataset as input for the > SimplePageRankComputation program. The dataset format looks like this:=
> 61578010 61147436
> 61578037 61147436
> 61578040 61147436
> (vertex id's, with pairs representing edges)
>
> When I run the command with
> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat, I get thi= s
> error :
> java.lang.IllegalArgumentException: checkClassTypes: vertex index type= s not
> assignable, computation - class org.apache.hadoop.io.LongWritable,
> VertexInputFormat - class org.apache.hadoop.io.IntWritable
>
> So I tried running the command with
> -vif org.apache.giraph.io.formats.LongLongNullTextInputFormat and I ge= t a
> different one:
> java.lang.IllegalArgumentException: checkClassTypes: vertex value type= s not
> assignable, computation - class org.apache.hadoop.io.DoubleWritable, > VertexInputFormat - class org.apache.hadoop.io.LongWritable
>
> I dont understand why the types in the input show up as different form= ats in
> each error. Also, as far as I could tell, there is no input format for=
> DoubleDouble. Is there a different way to get the graph into Giraph wi= thout
> having to write custom input code ? Thoughts would be much appreciated= .
>
> -----
> Reference Command:
> hadoop jar giraph-examples-1.1.0-for-hadoop-1.1.2-jar-with-dependencie= s.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.PageRankComputation -vif
> org.apache.giraph.io.formats.LongLongNullTextInputFormat -vip
> /user/kenrick/twitter/input -op /user/kenrick/twitter/output -w 30
> -----
>
> Thanks,
> Kenrick





--047d7b5d9acded2993051544b897--