Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 25AD0D2D7 for ; Mon, 20 Aug 2012 20:53:00 +0000 (UTC) Received: (qmail 28133 invoked by uid 500); 20 Aug 2012 20:53:00 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 28093 invoked by uid 500); 20 Aug 2012 20:52:59 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 28085 invoked by uid 99); 20 Aug 2012 20:52:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Aug 2012 20:52:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of countkaushik@gmail.com designates 209.85.220.180 as permitted sender) Received: from [209.85.220.180] (HELO mail-vc0-f180.google.com) (209.85.220.180) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Aug 2012 20:52:52 +0000 Received: by vcbfw7 with SMTP id fw7so5673128vcb.11 for ; Mon, 20 Aug 2012 13:52:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=oPxcbx4HiJElxOsHVFUuzbBcOHIslxlOs7DeCA/rZHA=; b=V0S+Ly4FiqwboYGiyMFAA0M1PxTP1eoSwvr7MtoCFb+KTEZHuQgTdrrLYym+cOFsZq kFivCaU7cZ4MmYD2Oa5OwQgpqEbPcST+xe6CGHz6MzGC3I8m3CwZjdoSSSmPvqbOHUpP ASGDOYB/GV32yKnO6SWhfgtYvprxWHKGRwM+TdtKuSHMpvxDAXxoXbL4TOH//dz/frVe cKYFcgq8tJ9fF2YIFm54Heg59PZZf5JpxzqKQPR0DsAuHVwyYSMK8SRLvq6r0wzDTkCQ k+O0MYqtPH6yApA5zXlyGxwBLFPTnvkicpIg+SgcCooXHWWI8lbuDi3LF9qwlsoE1bGa 4pSQ== MIME-Version: 1.0 Received: by 10.58.133.48 with SMTP id oz16mr12536116veb.1.1345495951428; Mon, 20 Aug 2012 13:52:31 -0700 (PDT) Received: by 10.220.67.10 with HTTP; Mon, 20 Aug 2012 13:52:31 -0700 (PDT) In-Reply-To: <2552BA18-DDCC-4F7E-973B-2812364E21AE@benchmarksolutions.com> References: <2552BA18-DDCC-4F7E-973B-2812364E21AE@benchmarksolutions.com> Date: Mon, 20 Aug 2012 13:52:31 -0700 Message-ID: Subject: Re: Adding MasterCompute object causes "failed to report status" errors From: KAUSHIK SARKAR To: user@giraph.apache.org Content-Type: multipart/related; boundary=047d7b6d7f423b4ab304c7b8b04b --047d7b6d7f423b4ab304c7b8b04b Content-Type: multipart/alternative; boundary=047d7b6d7f423b4ab104c7b8b04a --047d7b6d7f423b4ab104c7b8b04a Content-Type: text/plain; charset=ISO-8859-1 Hi Nick, Are you using WorkerContext to register the aggregator? You need to override the preApplication() method in WorkerContext to register the aggregator and then override the preSuperstep() method to to tell the workers to use the aggregator (the useAggregator() method). Check the MasterCompute and WorkerContext examples in Giraph. Regards, Kaushik On Mon, Aug 20, 2012 at 1:26 PM, Nick West wrote: > Hi, > > I have a giraph application that runs fine; however, when I add a > MasterCompute object (definition following) all of the map tasks time out. > I have hadoop configured to run with 8 map processes and giraph to use one > worker. > > Here's the definition of the MasterCompute object: > > class BPMasterComputer extends MasterCompute{ > override def compute() { > val agg = > getAggregator("VOTE_TO_STOP_AGG").asInstanceOf[BooleanAndAggregator] > val res = agg.getAggregatedValue.get > if (res) haltComputation > agg.setAggregatedValue(true) > } > override def initialize() { > registerAggregator("VOTE_TO_STOP_AGG", classOf[BooleanAndAggregator]) > val agg = > getAggregator("VOTE_TO_STOP_AGG").asInstanceOf[BooleanAndAggregator] > agg.setAggregatedValue(true) > } > override def write(out: DataOutput) {} > override def readFields(in: DataInput) {} > } > > (as far as I can tell, there is no state that needs to be read/written.) > I then register this class as the MasterCompute class in the giraph job: > > job.setMasterComputeClass(classOf[BPMasterComputer]) > > and then use the aggregator in the compute method of my vertices: > > class BPVertex extends EdgeListVertex[IntWritable, WrappedValue, Text, > PackagedMessage] with Loggable { > override def compute(msgs: java.util.Iterator[PackagedMessage]) { > ... > var stop = false > val agg = > getAggregator("VOTE_TO_STOP_AGG").asInstanceOf[BooleanAndAggregator] > ... code to modify stop and vote to halt ... > agg.aggregate(stop) > } > } > > Is there some other method that I am not calling that I should? Or some > step that I'm missing? Any suggestions as to why/how these additions are > causing the processes to block would be appreciated! > > Thanks, > *Nick West > ** > *Benchmark Solutions > 101 Park Avenue - 7th Floor > New York, NY 10178 > Tel +1.212.220.4739 | Mobile +1.646.267.4324 > *www.benchmarksolutions.com * > *** > > > > * > ** > --047d7b6d7f423b4ab104c7b8b04a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Nick,

Are you using WorkerContext to register the agg= regator? You need to override the preApplication() method in WorkerContext = to register the aggregator and then override the preSuperstep() method to t= o tell the workers to use the aggregator (the useAggregator() method). Chec= k the MasterCompute and WorkerContext examples in Giraph.

Regards,
Kaushik

On Mon, Aug 20, 2012 at 1:26 PM, Nick West <nick.west= @benchmarksolutions.com> wrote:
Hi,

I have a giraph application that runs fine; however, when I add a MasterCom= pute object (definition following) all of the map tasks time out. I have ha= doop configured to run with 8 map processes and giraph to use one worker. = =A0

Here's the definition of the MasterCompute object:

class BPMasterComputer extends MasterCompute{
=A0 override def compute() {
=A0 =A0 val agg =3D getAggregator("VOTE_TO_STOP_AGG").asInst= anceOf[BooleanAndAggregator]
=A0 =A0 val res =3D agg.getAggregatedValue.get
=A0 =A0 if (res) haltComputation
=A0 =A0 agg.setAggregatedValue(true)
=A0 }
=A0 override def initialize() {
=A0 =A0 registerAggregator("VOTE_TO_STOP_AGG", classOf[Boole= anAndAggregator])
=A0 =A0 val agg =3D getAggregator("VOTE_TO_STOP_AGG").asInst= anceOf[BooleanAndAggregator]
=A0 =A0 agg.setAggregatedValue(true)
=A0 }
=A0 override def write(out: DataOutput) {}
=A0 override def readFields(in: DataInput) {}
}

(as far as I can tell, there is no state that needs to be read/written= .) =A0I then register this class as the MasterCompute class in the giraph j= ob:

job.setMasterComputeClass(classOf[BPMasterComputer])

and then use the aggregator in the compute method of my vertices:

class BPVertex extends EdgeListVertex[IntWritable, WrappedValue, Text,= PackagedMessage] with Loggable {
=A0=A0 override def compute(msgs: java.util.Iterator[PackagedMessage])= {
=A0 =A0 ...
=A0 =A0 var stop =3D false
=A0 =A0 val agg =3D getAggregator("VOTE_TO_STOP_AGG").asInst= anceOf[BooleanAndAggregator]
=A0 =A0 ... code to modify stop and vote to halt ...
=A0 =A0 agg.aggregate(stop)
=A0 }
}

Is there some other method that I am not calling that I should? =A0Or = some step that I'm missing? =A0Any suggestions as to why/how these addi= tions are causing the processes to block would be appreciated!

Thanks,
Nick West

Benchmark Solutions
101 Park Avenue - 7th Floor
New York, NY 10178=A0
Tel +1.212.220.4739 | Mobile=A0+1.646.267.4324=A0
www.benchmarksolutions.= com=A0
= =A0
=A0


--047d7b6d7f423b4ab104c7b8b04a-- --047d7b6d7f423b4ab304c7b8b04b Content-Type: image/png; name="image001.png" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: 5a1969ae57eea6c4_0.1 iVBORw0KGgoAAAANSUhEUgAAAPQAAAAmCAYAAADz2bbaAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAZdEVYdFNvZnR3YXJlAE1pY3Jvc29mdCBPZmZp Y2V/7TVxAAALC0lEQVR4Xu2dsYskRRTG7z9RMFYwVjBWTEUxEkERMVMvPsVEk0MOE+FEowNPxECD OwRBhFMDMfAOExE5Iw+DAwWzkd/AN3z7zavqnt3Z3dmdahhuu7uquurV+9733qvqvguLcQwJDAmc GwlcODcjGQMZEhgSWBwa0L//cW/xy69/LvTvkOWQwJDA6UtgNqD/+fe/xadf/LC49O7nizcvXVs8 8+KV5d8fX/t2de2b7+4s3rvy1eKve/dPf2SjB0MCeyiBWYAGqAAXRv7y5s9LMANu/v7g6tdLEOsA zJTnOuXHMSQwJHByEugCWiB+9eInSxAD0Bde+3DJ0DC2GJlrnPtBXYE6753c8MaThgT2SwJNQAuQ iAPgAmoYGWaGkQExIOc6ZSnDfRga4KsM4JcB2C/RjtEOCZy8BNYADSCJjQGrfu5S08Uff/ptCVj+ 1cE55blGfQBPwgwDIBf95Ic3njgksF8SWAM0bAsAxbhVLMw9JcVgZRhZ4Me9Bsi6jzj5G4DT1nk9 7t+/v/j+1q3V7yyP887t26tx3L17d6eH4jJnDvb9OABowKysNYKBXQEi13TA4JwrrnYm5xpgVl0Z Bk+iYQzO44FiPfTAg6vfWR7j888+txrH+5cvb20obiiQ1zYOl/m22mz167jksg05qI0DgAa8/GBd X5rCneZc7A1wOQC3st4ObMoq7oaVFX9zDSPRO1xoPln6+5WXXl7cvHFjmzLYSlsD0LUYYc133np7 8ejDjxwweJrPi6+/cST5HxegK8/kzAFaSS0ltDjnEBvDvkqGiWkBtcDshkDsrNkC2HK9e1nvKUBr AlGSXToGoNdnAzA//eRTJZA1j8z3UY5tApr+fnT16qrP2a8zA2gA5vEtMbE2h5DkAohKjCnJJZYW Q5MI00Fd2pP7LTbXphN34XtCw6ojRH5PPPb4mmLsUsw0AL0OS9x1BxxzyDV+GGTAflTDvE1AT83h WcgtLF1uuc7OprjNgFXZazEy4HeWlmsuNteOMsqkey133sHfA3Ra71SQVswkwW+a0FGCZYoxaFdl ZVRayqBkGX2ac6jtylipralxef/mPJMy/txtMZG3g3He1ACrTz3ZnSSg58jSk6NzynsZ6e1UPele JZcloLUDTA1VCS9nXK1Fa82Zcy1RCezKcmu3GID3dltutytBAjpB44BGkMRjGXeLFVxIXoZ4vKpX MQfuWOUpENdn3xB2hg/UVfw/pzz1GVerLZ9Q2qUfrbyDl/UytO/16FcL0DzD6065yx43T5XV/Gge M+bmnHlKY1YB2q9lUq/SoakwT33vGbrPrl8vdYM6STzeDvfoY46XsboBrHRA4+TZOpaAhoX5AUoY F6BqPdmTXdpI4uvPasiXrmBgzuWua11bS2G003K7e4BO4GnAc2I1T774hLeSNZRxZahA73FgKkqv XTHPnH7glrbawkDoSO8lgU0bAoPfw3AlKCrFRaG8H/RrinETKBjE3pHPqIwTfWgZJwHnpAHdMqSt frhcKoLIhOGUXFxPl4BWMkvxsNxnJbK0a0zs23r5QltF3XXXllHYGWPB4TF5z+X2GDqV2q2Sg41y KI4s35QF577Hdv4cAYZnJfgAgmJB7iegKc/EUSaNAedVeRSD8lUiSfdSeaTEsvL0CyblOnLw8cjr qICiaxVDp8F049ADaGVkJOsqbEjlRk6SccpfxmRqfucwNPNXzZPifelaZejSIGrOqZvjkXeWhg55 6lmp55XnSVnmifY03wcYWuzKRYCnhJdcZHePe+vI3HP2Vn3aVTJN+8E9YeZKMeX+MGAHMwPuxVEu cLF0KocrFyBIl9QBlgyhvidA08VMzyPLA1QdWGTvgzNxjlcKS3sVY7oxUp8S0M6ctJGK60akNf4W sHvz6W5lGs1k85SX7m8D0K057JENcs+5yDnnvoMaPeJImbhLnmFNGli10ZL3kqG1VVNsDYNm9lqZ a+61GFoAhsmVTFN2XMtbPItfaz06EymcV26J2MYnGmVTwkD/OkAlDFeCXAetYqx0USthZr1cK3e2 Ulzl7Wb5ntvo8nAGUkJG1j5ZpwJ0pSA5B94XN6Zd/9lutuJL2pXr7kbDDZg/ww1rNZbDutyHBXSC r0pSVQTh8q3kn0YqPTx5Oc2kmMCLS6y4V0ksgKcdY4qnq/gX5q12kGmnGYwMg8P2eu2yUoheDJ3C QVHyWs+d5B6Hl5lignQbW5taEtDJllOAzsRJD+yV61clVlIWFQiqnWAtVk0GmgtolUN2Vdv0oTfv qu/eVmWcTxrQqRtzDH0v6aj6CWg8yFYeBbm5rq2y3Nolphcs1Lh2h3nWGlDCxgBdL3P4K5Tay50J NWW8ud9i+amJTSt9VEAnkCrXrgeuudb9KIDuZUlpN2WABVeM79b9MIBORaoMwKbATveafk3NO89w Ft8Fhk5AVyHPVFa9kmcVRgBq5rLyVt3QLgENQHGD5VZ7zMw13Gb/WIFezlCSS6623GgBWplzGNnX njmfE0NXjJCATpd7jnL1Yu4pl7u1VTHrZT+OE9AuE4/F6cNUDD3F0MmeyC4NzByZZxlXTOY5k30V OLJOeltzGDqNiY9lag7TM0qXu/LeMmlW5ShSNj39pCz9zMSp2ljt5YaJcbcBZya+9KaUQK34WNlw AVws7W9s8SDq+U60uTvFHNDalueDRQkyMdFaGvGJ2xTQ6SZWscuUMhwnoOe657BtgmAOoNPlm5vl bm2USFAxj3kt+9VKmlVz2Vr/ziRVGqecw8zEJ6Bpr7eUV91H/lXI5KCeAjRl0ytbAzSsLBcZNhWD AlbfLOKbQ9xtdpaW+643s7R+rTi792miKinWih9kETNpQBsogNL6suytGMWFWTF0XtMmByWdlIzz iTgthmasTDbjr+LVwwA6mV7JrClmlhGjT/RF2z0ztleiLVmHOtVyEu31lq1SH7TkV7mrPYamP0ow toCYbrcSVtULKXrWpoDmGeoLbWS+RIaaPh542wqG1u4vrRUrpgbY+eEDQAzzUk9vV2nziH/VhL+V ZKONbbyckUstUy8BKCHWctOknBWguZeuUyrlaTJ0un7et8o162XQewySbvHUm1Kp7FXC0tuAEXsb Lag/Z2NJLvv5c5McMnyoni/AtIDY23SkZ/vqwGEA3Uv2ru0UkzKLQZX11poxwBUItdlEe7jF6lq7 VlsYBq6JjeXKy01vWXeAoyRJ9S9K0trL7G/KpFK7C+ftpvusLXYqk9srK8OBEmS9HJ8YU0zVe45A VfVBxkX3NJlVXKV1Xh9vtl0tQ/kc5Jo/oO7Jz8fdymozN9oum3KCeZmrBBbnuR2yJyeeXW1UUQzb ki26lZ6NAN2SizyY1oagNBq9dqoxtZb9KhmufbFEgAWoHP6VT859txggh52pk+vKAJqy/tnf/JTR lMs27g8JDAlsJoHyI4ECMUBVltu/PqLstt6sgn3z9Un/Xrf2bo+vf242OaP0kMCmEigBDfAAre/2 Atj6dK8+EujuNgxNHTG8XHDamPpKyaadHuWHBIYEagk0P+Pr7rTcb29C3x9TNlwJM871N0CmHMZA ybbxv2oMVRwSOD4JTP7PGSS1AKb+Hyt1xfdka91ZG1CU9VZZvY6p96aPbzij5SGB/ZbAJKAdlFqi wp3W10fcLde+78HC+61UY/SnJ4HZgM4uwthi7dY2ztMb1njykMB+SuDQgN5PcY1RDwnstgQGoHd7 fkbvhgQ2ksAA9EbiGoWHBHZbAgPQuz0/o3dDAhtJAED/PX5DBkMHzocO/A8padvYOoPF2AAAAABJ RU5ErkJggg== --047d7b6d7f423b4ab304c7b8b04b--