Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4778E11933 for ; Mon, 21 Jul 2014 22:28:53 +0000 (UTC) Received: (qmail 39715 invoked by uid 500); 21 Jul 2014 22:28:53 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 39663 invoked by uid 500); 21 Jul 2014 22:28:53 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 39650 invoked by uid 99); 21 Jul 2014 22:28:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2014 22:28:52 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [207.46.163.182] (HELO na01-bn1-obe.outbound.protection.outlook.com) (207.46.163.182) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2014 22:28:48 +0000 Received: from BLUPR04MB820.namprd04.prod.outlook.com (10.141.209.155) by BLUPR04MB580.namprd04.prod.outlook.com (10.141.202.28) with Microsoft SMTP Server (TLS) id 15.0.990.7; Mon, 21 Jul 2014 22:28:26 +0000 Received: from BLUPR04MB820.namprd04.prod.outlook.com (10.141.209.155) by BLUPR04MB820.namprd04.prod.outlook.com (10.141.209.155) with Microsoft SMTP Server (TLS) id 15.0.990.7; Mon, 21 Jul 2014 22:28:24 +0000 Received: from BLUPR04MB820.namprd04.prod.outlook.com ([10.141.209.155]) by BLUPR04MB820.namprd04.prod.outlook.com ([10.141.209.155]) with mapi id 15.00.0990.007; Mon, 21 Jul 2014 22:28:24 +0000 From: "Sardeshmukh, Vivek" To: "user@giraph.apache.org" Subject: RE: Setting variable value in Compute class and using it in the next superstep Thread-Topic: Setting variable value in Compute class and using it in the next superstep Thread-Index: AQHPpSPTtmCb3xSzz0CzOy003rNTzZurDUMGgAAJcYCAAAHigIAAAJYf Date: Mon, 21 Jul 2014 22:28:24 +0000 Message-ID: <1405981703655.95944@uiowa.edu> References: <1405976759238.96647@uiowa.edu> <9B590560869E3741B7665052119D02421FC0F5@DEN-EXDDA-S52.corp.ebay.com>,,<9B590560869E3741B7665052119D02421FC22B@DEN-EXDDA-S52.corp.ebay.com> In-Reply-To: <9B590560869E3741B7665052119D02421FC22B@DEN-EXDDA-S52.corp.ebay.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.255.45.18] x-microsoft-antispam: BCL:0;PCL:0;RULEID: x-forefront-prvs: 0279B3DD0D x-forefront-antispam-report: SFV:NSPM;SFS:(377454003)(24454002)(52034003)(189002)(199002)(76176999)(21056001)(54356999)(83072002)(105586002)(92566001)(36756003)(4396001)(106356001)(79102001)(107886001)(110136001)(99286002)(106116001)(75432001)(87936001)(95666004)(19627405001)(2351001)(99396002)(92726001)(85852003)(107046002)(89122001)(2656002)(101416001)(31966008)(81542001)(64706001)(85306003)(74662001)(83322001)(86362001)(81342001)(20776003)(50986999)(74502001)(80022001)(77982001)(19580405001)(93886003)(66066001)(19617315012)(15202345003)(19580395003)(16236675004)(15975445006)(88552001)(76482001)(46102001)(19627315001)(19607625011);DIR:OUT;SFP:;SCL:1;SRVR:BLUPR04MB820;H:BLUPR04MB820.namprd04.prod.outlook.com;FPR:;MLV:sfv;PTR:InfoNoRecords;MX:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_140598170365595944uiowaedu_" MIME-Version: 1.0 X-Microsoft-Antispam: BCL:0;PCL:0;RULEID: X-OriginatorOrg: uiowa.edu X-Virus-Checked: Checked by ClamAV on apache.org --_000_140598170365595944uiowaedu_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you Matthew for the example link. It is helpful. I'll give it a shot. If I have a custom vertex class isn't it necessary to change the VertexInpu= tFormat class too? Since this class "loads" the data into the vertex and if= vertex has a custom value field then it doesn't know how to load the input= . Am I right? Vivek ________________________________ From: Schweiger, Tom Sent: Monday, July 21, 2014 5:16 PM To: user@giraph.apache.org Subject: RE: Setting variable value in Compute class and using it in the ne= xt superstep For more than one flag, a custom class is necessary (unless you're able to,= say, toggle the sign bit to get double usage out or a value). I've started a private thread with Vivek to get a better understanding of w= hat he was trying to solve. And you are also correct that there isn't much to writing a custom vertex c= lass. The key is making sure you read and write in the same order. Likewi= se, extending a vertex reader can be quite simple. ________________________________ From: Matthew Saltz [saltzm@gmail.com] Sent: Monday, July 21, 2014 3:09 PM To: user@giraph.apache.org Subject: Re: Setting variable value in Compute class and using it in the ne= xt superstep Tom, If it's necessary to store more than one flag though, for example, won't a = custom class be necessary? I'm a beginner too, so I apologize if I'm incorr= ect about that. Just to be clarify, to keep persistent data for a vertex fr= om one superstep to the next, it is necessary to encapsulate it in the type= used for the 'V', right? In other words, if Vivek tries to use a normal me= mber variable for the Computation class, it won't work will it? Also, just to point out, there actually isn't too much involved with writin= g your own custom vertex class. Here's a quick example to get you started. Within your compute() = method you can access the data in this class by doing SampleVertexData d =3D vertex.getValue(); and then using d.setFlag(true) or boolean currentFlag =3D d.getFlag() for e= xample. And your computation class is now something like public class MyComputation extends BasicComputation { @Override public void compute(Vertex vertex, = Iterable messages) {.....} ... } As a warning, for this class I'm using Hadoop 0.20.203 and I'm also a begin= ner, so take everything I say with a grain of salt, and Tom please correct = me if I'm wrong. Best of luck, Matthew On Mon, Jul 21, 2014 at 11:37 PM, Schweiger, Tom > wrote: And in answer of : This post also suggests (along with what I described above) to have a field= in the vertex value itself. For that I need to change the vertex input for= mat and also create my own custom vertex class. Is it really necessary? No, you don't need a custom vertex class or vertex input format. You can cr= eate/initialize the value at the beginning of the first superstep. ________________________________ From: Sardeshmukh, Vivek [vivek-sardeshmukh@uiowa.edu] Sent: Monday, July 21, 2014 2:05 PM To: user@giraph.apache.org Subject: Setting variable value in Compute class and using it in the next s= uperstep Hi, all-- In my algorithm, I need to set a flag if certain conditions hold (locally a= t a vertex v). If this flag is set then execute some other block of code *o= nly once*, and do nothing until some other condition is hold. My question is, can I declare a flag variable in the class where I override= compute function? I defined the flag as a public variable and setting it o= nce the conditions are met but it seems the value is not "carried" over to = the next superstep. I dig a little bit in this mailing list and found this https://www.mail-archive.com/user@giraph.apache.org/msg01266.html This post also suggests (along with what I described above) to have a field= in the vertex value itself. For that I need to change the vertex input for= mat and also create my own custom vertex class. Is it really necessary? By the way, I am using Giraph 1.1.0 compiled against Hadoop 1.0.3. I was ab= le to run SimpleShortestPathComputation successfully. Here are more technical details of my algorithm: I am trying to implement D= elta-stepping shortest path algorithm ( http://dl.acm.org/citation.cfm?id= =3D740136 or http://citeseerx.ist.psu.edu/viewdoc/summary?doi=3D10.1.1.46.2= 200 ). This was mentioned in Pregel paper. A vertex "relax" light edges if = it belongs to the minimum bucket index (of course, aggregators!). Once a ve= rtex is done with relaxing light edges it relaxes heavy edges (here is wher= e I need a flag) once. A vertex may be "re-inserted" to a newer bucket and = may have to execute all the steps that I described here again. Thanks. Sincerely, Vivek A beginner in Giraph (and Java too!) --_000_140598170365595944uiowaedu_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Thank you Matthew for the example link. It is helpful. I'll give it= a shot.


If I have a custom vertex class isn't it necessary to change the VertexI= nputFormat class too? Since this class "loads" the data into the = vertex and if vertex has a custom value field then it doesn't know how to l= oad the input. Am I right?



Vivek
From: Schweiger, Tom <t= hschweiger@ebay.com>
Sent: Monday, July 21, 2014 5:16 PM
To: user@giraph.apache.org
Subject: RE: Setting variable value in Compute class and using it in= the next superstep
 

For more than one flag, a custom class is necessary (unless you're able to,= say, toggle the sign bit to get double usage out or a value).

I've started a private thread with Vivek to get a better understanding of w= hat he was trying to solve.

And you are also correct that there isn't much to writing a custom vertex c= lass.  The key is making sure you read and write in the same order.&nb= sp; Likewise, extending a vertex reader can be quite simple.

From: Matthew Saltz [saltzm@gmail.com]
Sent: Monday, July 21, 2014 3:09 PM
To: user@giraph.apache.org
Subject: Re: Setting variable value in Compute class and using it in= the next superstep

Tom, 

If it's necessary to store more than one flag though, for example, won= 't a custom class be necessary? I'm a beginner too, so I apologize if I'm i= ncorrect about that. Just to be clarify, to keep persistent data for a vert= ex from one superstep to the next, it is necessary to encapsulate it in the type used for the 'V', right? In = other words, if Vivek tries to use a normal member variable for the Computa= tion class, it won't work will it?

Also, just to point out, there actually isn't too much involved with writin= g your own custom vertex class. Here's a quick example to get you started. Within your compute() method yo= u can access the data in this class by doing 

SampleVertexData d =3D vertex.getValue();

and then using d.setFlag(true) or boolean currentFlag =3D d.getFlag() = for example.  And your computation class is now something like

public class MyComputation extends BasicComputation<IdType, = SampleVertexData, EdgeType, MessageType> {
    @Override
    public void compute(Vertex<IdType, SampleVe= rtexData, EdgeType> vertex, Iterable<MessageType> me= ssages) {.....} 
    
    ...

}

As a warning, for this class I'm using Hadoop 0.20.203 and I'm also a = beginner, so take everything I say with a grain of salt, and Tom please cor= rect me if I'm wrong. 

Best of luck,
Matthew


On Mon, Jul 21, 2014 at 11:37 PM, Schweiger, Tom= <thschweiger@e= bay.com> wrote:
And in answer of :


This post a= lso suggests (along with what I described above) to have a field in the ver= tex value itself. For that I need to change the vertex input format and also create my own custom vertex class. Is it = really necessary?
No, you don't need a custom vertex class or vertex input format. You can cr= eate/initialize the value at the beginning of the first superstep.

Fro= m: Sardeshmukh, Vivek [vivek-sardeshmukh@uiowa.edu]
Sent: Monday, July 21, 2014 2:05 PM
To: user= @giraph.apache.org
Subject: Setting variable value in Compute class and using it in the= next superstep

Hi, all--


In my algorithm, I need to set a flag if certain conditions hold (locall= y at a vertex v). If this flag is set then execute some other block of code= *only once*, and do nothing until some other condition is hold.


My question is, can I declare a flag variable in the class where I overr= ide compute function? I defined the flag as a public variable and setting i= t once the conditions are met but it seems the value is not "carried&q= uot; over to the next superstep.

I dig a little bit in this mailing list and found this

https://www.mail-archive.com/user@giraph.apache.org= /msg01266.html


This post also suggests (along with what I described above) to have a fi= eld in the vertex value itself. For that I need to change the vertex input = format and also create my own custom vertex class. Is it really necessary?


By the way, I am using Giraph 1.1.0 compiled against Hadoop 1.0.3. I was= able to run SimpleShortestPathComputation successfully.


Here are more technical details of my algorithm: I am trying to implemen= t Delta-stepping shortest path algorithm ( ht= tp://dl.acm.org/citation.cfm?id=3D740136 or http://citeseerx.ist.psu.edu/viewdoc/summary?doi=3D10.1.1.46.2200 ). This was mentioned in Pregel paper. A vertex "relax" light ed= ges if it belongs to the minimum bucket index (of course, aggregators!). On= ce a vertex is done with relaxing light edges it relaxe= s heavy edges (here is where I need a flag) once. A ver= tex may be "re-inserted" to a newer bucket and may have t= o execute all the steps that I described here again. 


Thanks.


Sincerely,

Vivek
A beginner in Giraph (and Java too!)


--_000_140598170365595944uiowaedu_--