Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7350E17E04 for ; Tue, 11 Nov 2014 19:37:45 +0000 (UTC) Received: (qmail 44750 invoked by uid 500); 11 Nov 2014 19:37:45 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 44694 invoked by uid 500); 11 Nov 2014 19:37:45 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 44683 invoked by uid 99); 11 Nov 2014 19:37:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Nov 2014 19:37:45 +0000 X-ASF-Spam-Status: No, hits=3.2 required=5.0 tests=FORGED_YAHOO_RCVD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of puagarwal@yahoo.com designates 98.136.218.192 as permitted sender) Received: from [98.136.218.192] (HELO nm6-vm1.bullet.mail.gq1.yahoo.com) (98.136.218.192) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Nov 2014 19:37:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1415734573; bh=T7HncDMgvLWeZnRbXAFegSrpZE2iYpKoe1gBNkQX4fA=; h=Date:From:Reply-To:To:In-Reply-To:References:Subject:From:Subject; b=iR39QjtGPUnedzh1MYe6JcC47mcOZtq2z2zWy664E8Ng/NZiys7TJ001BD2IffhV44ZolHxi3Jr5q8MsstfGaWZOV8EagErX/stAzxFAsBh1FxE6tifaYzSuXAYeILLAOFe0MUm2a6gHPBJPc7Or/8o/rGu+yw7SPWj5M7KBa/iFWyUnL//+KmT0XicaeMzEZpdRlC0UAr0E0Dafs0XawUcB2eQhcURt9prwUCjAASjo7q9Vb23lKngK4UH6fFP+zcPGD4PIr6TIpRSh6TOSR39liRcwwDmE2JXUWMKtdGDEEXEbieIBtBJPjmWvBYcMJ6o98wbl2WcDn9Vu/i6CoQ== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048; d=yahoo.com; b=sI+p1NhnS5OvdW5DS+nHNtfS9YExFehmSasaZHt+r6XEZf1O1VDjJPtnQFIuFou2fhQYrE3V4vpSsjcBngd7swyCkZu8VdF+pwb9TrHW/moOB/o6v2+ysSZzHzH9NEMH4IR1WW+VDdLvlqUnRTucJTxtntQzxtuORSuf04gJQY/q0FUpVn8jAJ7tUaObvn6Jjw2vf/B9LE6EupEHVEHRXZFuH+QbkW5Gm7f0k4osUkqjZcMeAdkTeuCX/6LEGX8H8ZZu5uNKiFbIFrxqB1S2+AXJchgpJfMATlidzHh3Vvggw7u/iDzIb0SGAdpcVKDc/2FLKOtKpaA/PTMiQjAW+g==; Received: from [98.137.12.63] by nm6.bullet.mail.gq1.yahoo.com with NNFMP; 11 Nov 2014 19:36:13 -0000 Received: from [98.137.12.237] by tm8.bullet.mail.gq1.yahoo.com with NNFMP; 11 Nov 2014 19:36:13 -0000 Received: from [127.0.0.1] by omp1045.mail.gq1.yahoo.com with NNFMP; 11 Nov 2014 19:36:13 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 54492.55790.bm@omp1045.mail.gq1.yahoo.com X-YMail-OSG: ioKr9wgVM1kEsm6Mw2aYWyb0Y4Nb1FkfcW5K2ZyNEh1LWq0ZmuyT9wSFjzxiR.Q EGkZBqtPoe9CRyA2sAWB2pM.PAkepWap9nKRYgP2BUPlPAVh6NIESb6DildBiAA4kA9eJew3JqkR p5Z0Fzr4WkA9U4HRG061odOqyZBu1F3yEbSGk7w4YbadPmBLPqTP2YKPkhjdVacTtVEIr_EhuSQQ a26k3alwC9RxthlODyfqPufARy9C3FSYRqmRC93bHh00uo5sHYMDoRFQfhcMBcEjQWhx8mmAA66F IkXq0wlGwEfBh2iqwgCOnhWwA0GcnxfuXSUsIWM2QnqaXo_RRtjcsR4185zRTzH0n8aEJS5N4lQ. hkSjbZOpY_PAYmE1oksFx2deGREBhF1z9m6ldj1QRCKiZEy4uKGog_3_K5V11JZwGfBQyzZCzAo6 DIo10qd0WO.TGeu_Pk_zqXRUL4wT_DLIaHKd5q_2ZrPjdaWpMgPSG0ZuwMDzKxIzekb8U Received: by 216.39.60.159; Tue, 11 Nov 2014 19:36:12 +0000 Date: Tue, 11 Nov 2014 19:36:12 +0000 (UTC) From: Puneet Agarwal Reply-To: Puneet Agarwal To: "user@giraph.apache.org" , Matthew Saltz , Puneet Agarwal Message-ID: <981333378.66992.1415734572306.JavaMail.yahoo@jws10782.mail.gq1.yahoo.com> In-Reply-To: <1199578669.83567.1415556897923.JavaMail.yahoo@jws10741.mail.gq1.yahoo.com> References: <1199578669.83567.1415556897923.JavaMail.yahoo@jws10741.mail.gq1.yahoo.com> Subject: Re: Issue in Aggregator MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_66991_1224952835.1415734572297" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_66991_1224952835.1415734572297 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I am stuck at this, requesting help again. Please inform how giraph aggregated the global value of an aggregator from = the various values itreceives from the local aggregators?does it call the s= ame class ?It seems not, please confirm. =20 On Sunday, November 9, 2014 11:48 PM, Puneet Agarwal wrote: =20 Dear Matthew, Yes my aggregator is commutative, and I want globally aggregated value from= all workers, and not the locally aggregated. But I am getting locally aggregated value. My query is :=20 Does Giraph call the aggregator class for the values returned by the local = aggregators or not? - PuneetIIT Delhi, India =20 On Sunday, November 9, 2014 5:43 AM, Matthew Saltz = wrote: =20 Hi Puneet,It's unclear to me what you're wanting in terms of aggregator be= havior. Are you saying you want an aggregator such that the final output is= the aggregated value just for a particular worker? With an aggregator you = should at least make sure the operations you're performing are commutative;= that is, the order in which items are aggregated should not matter unless = it is explicitly dealt with somehow. Otherwise you'll get unpredictable res= ults. Best, Matthew SaltzEl 08/11/2014 15:05, "Puneet Agarwal" es= cribi=C3=B3: Hi All,In my algo, I use an Aggregator which takes a Text value. I have wri= tten my custom aggregator class for this, as given below. public class MyAgg extends BasicAggregator {...} This works fine when running on my laptop with one worker.However, when run= ning it on the cluster, sometimes it does not return the correctly aggregat= ed value.It seems it is returning the locally aggregated value of one of th= e workers.While it should have used my logic to decide which of the aggrega= ted values sent by various worker should be chosen as finally aggregated va= lues.(But in fact I have not written such a code anywhere, it is therefore = doing the best it could) Following is how is my analysis about this issue.a.=C2=A0=C2=A0=C2=A0=C2=A0= I guess every worker aggregates the values locally.b.=C2=A0=C2=A0=C2=A0 the= n there is a global aggregation step, which simply compares the values sent= by various aggregators.c.=C2=A0=C2=A0=C2=A0 For global aggregation it uses= Text.compareTo() method. This method Text.compareTo() is a default Hadoop = implementation and does not include the logic of my program.d.=C2=A0=C2=A0= =C2=A0 It seem it is because of the above the value returned by my aggregat= or in the cluster is actually not globally aggregated, but the locally aggr= egated value of one of the worker gets taken. If the above analysis is correct, following is how I think I can solve this= .I should write my own class that implements Writable interface. In this cl= ass I would also write a compareTo method as a result things will start wor= king fine. If it was using class MyAgg itself, to decide which of the values returned = by various workers should be taken as globally aggregated value then this p= roblem would not have occurred. I seek your guidance whether my analysis is correct. - PuneetIIT Delhi, India =20 ------=_Part_66991_1224952835.1415734572297 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I am stuck at thi= s, requesting help again.

Pleas= e inform how giraph aggregated the global value of an aggregator from the v= arious values itreceives from the local aggregators?
does it call the same class ?
<= div id=3D"yui_3_16_0_1_1415734446255_5912" dir=3D"ltr">It seems not, please= confirm.


On Sunday, November 9, 2014 11:48 PM, Puneet Agar= wal <puagarwal@yahoo.com> wrote:


Dear Matthew,<= /div>

Yes my aggregator is commutative, and I want = globally aggregated value from all workers, and not the locally aggregated.=

But I am getting locally aggregated value.

My query is :
Does Giraph call the aggregato= r class for the values returned by the local aggregators or not?
- Puneet
IIT Delhi, India


On Sunday, November 9, 2014 5:43 AM, Ma= tthew Saltz <saltzm@gmail.com> wrote:

Hi Puneet,
It's unclear to me what you're wanting in terms of aggrega= tor behavior. Are you saying you want an aggregator such that the final out= put is the aggregated value just for a particular worker? With an aggregato= r you should at least make sure the operations you're performing are commut= ative; that is, the order in which items are aggregated should not matter u= nless it is explicitly dealt with somehow. Otherwise you'll get unpredictab= le results.
Best,
Matthew Saltz
El 08/11/2014 15:05, "Puneet Agarwal" &= lt;puagarwal@yahoo.com= > escribi=C3=B3:
Hi All,
In my algo, I use an A= ggregator which takes a Text value. I have written my custom aggregator cla= ss for this, as given below.

public class MyAgg extends BasicAggregator<Text= > {
...
}
=

This works= fine when running on my laptop with one worker.
Howe= ver, when running it on the cluster, sometimes it does not return the corre= ctly aggregated value.
It seems it is returning the l= ocally aggregated value of one of the workers.
While = it should have used my logic to decide which of the aggregated values sent = by various worker should be chosen as finally aggregated values.
(But in fact I have not written such a code anywhere, it is the= refore doing the best it could)
Following is how is my analysis abo= ut this issue.
a.    I guess ever= y worker aggregates the values locally.
b.  = ;  then there is a global aggregation step, which simply compares the = values sent by various aggregators.
c.  &nb= sp; For global aggregation it uses Text.compareTo() method. This method Tex= t.compareTo() is a default Hadoop implementation and does not include the l= ogic of my program.
d.    It seem it i= s because of the above the value returned by my aggregator in the cluster i= s actually not globally aggregated, but the locally aggregated value of one= of the worker gets taken.

<= div dir=3D"ltr">If the above analysis is correct, following is how I think = I can solve this.
I should write my own class that im= plements Writable interface. In this class I would also write a compareTo m= ethod as a result things will start working fine.
If it was using class MyAgg itself, = to decide which of the values returned by various workers should be taken a= s globally aggregated value then this problem would not have occurred.

I seek your guidance whether my analysis is correct.

- P= uneet
I= IT Delhi, India





=
------=_Part_66991_1224952835.1415734572297--