Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A5F9C904F for ; Thu, 16 May 2013 21:11:43 +0000 (UTC) Received: (qmail 67057 invoked by uid 500); 16 May 2013 21:11:43 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 66938 invoked by uid 500); 16 May 2013 21:11:43 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 66928 invoked by uid 99); 16 May 2013 21:11:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2013 21:11:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (athena.apache.org: error in processing during lookup of prvs=8848c8209a=majakabiljo@fb.com) Received: from [67.231.153.30] (HELO mx0a-00082601.pphosted.com) (67.231.153.30) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2013 21:11:39 +0000 Received: from pps.filterd (m0004003 [127.0.0.1]) by mx0b-00082601.pphosted.com (8.14.5/8.14.5) with SMTP id r4GL8qKs015568 for ; Thu, 16 May 2013 14:11:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=from : to : subject : date : message-id : in-reply-to : content-type : mime-version; s=facebook; bh=lacPZvprlciYBLrIa3Ytnr7m+LZzNOt7WOHB2ROVXdM=; b=eh+7g6S6qvmkwlflqagxWS1R2OKtdF/JtFMxoQ8+RzOeVnMJ58FAjI16OkOYRkAZWtOj E2DWMizlJ6KMLLXpGYdvF2tP0B7TyAYdstpXMQv9gC8po8GDFOVrOhhGPP1SNAmR01SP PIfk0sjA/r2L3fOusdlIUeQM5C4CpHhWZag= Received: from mail.thefacebook.com (prn1-cmdf-dc01-fw1-nat.corp.tfbnw.net [173.252.71.129] (may be forged)) by mx0b-00082601.pphosted.com with ESMTP id 1cd0ma0yph-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK) for ; Thu, 16 May 2013 14:11:17 -0700 Received: from PRN-MBX01-2.TheFacebook.com ([169.254.4.207]) by PRN-CHUB06.TheFacebook.com ([fe80::f073:2a60:c133:4d69%12]) with mapi id 14.02.0328.011; Thu, 16 May 2013 14:11:15 -0700 From: Maja Kabiljo To: "user@giraph.apache.org" Subject: Re: Broadcast of large aggregated value is slow. Thread-Topic: Broadcast of large aggregated value is slow. Thread-Index: Ac5SduFYj31MKEEOTxu2axxsg/H+mQAPB8QA//+NyAA= Date: Thu, 16 May 2013 21:11:13 +0000 Message-ID: <1F592C080E9ACB4CB1C9EA1865BF3EFA0D19D534@PRN-MBX01-2.TheFacebook.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.16.4] Content-Type: multipart/alternative; boundary="_000_1F592C080E9ACB4CB1C9EA1865BF3EFA0D19D534PRNMBX012TheFac_" MIME-Version: 1.0 X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626,1.0.431,0.0.0000 definitions=2013-05-16_06:2013-05-16,2013-05-16,1970-01-01 signatures=0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_1F592C080E9ACB4CB1C9EA1865BF3EFA0D19D534PRNMBX012TheFac_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Eric, Can you please take a look at the logs of one of the workers listed (13, 34= , 38, 50, 48, 52, 58, 56), what are they doing? The fact that a worker is w= aiting on aggregator can have different causes, it doesn=92t necessarily me= an that sending aggregators is slow. It can for example mean that some work= ers finished computing before others and are now waiting for others to fini= sh and send their data. How big are aggregators which you are using? Thanks, Maja From: Eric Kimbrel > Reply-To: "user@giraph.apache.org" > Date: Thursday, May 16, 2013 2:00 PM To: "user@giraph.apache.org" > Subject: Re: Broadcast of large aggregated value is slow. >From the attached logs in original post, you can see that both workers use = about 4 seconds of compute time on super step 4, but they complete super st= ep 4 about 10 minutes apart. Eric Kimbrel Software Engineer I Data Fusion & Analytics Sotera Defense Solutions, Inc. o: 360-516-6621 c: 360-990-1873 e: Eric.Kimbrel@soteradefense.com w: www.potomacfusion.com | www.soteradefense.com Agility. Ingenuity. Integrity. From: Eric Kimbrel > Reply-To: "user@giraph.apache.org" > Date: Thursday, May 16, 2013 1:50 PM To: "user@giraph.apache.org" > Subject: Broadcast of large aggregated value is slow. I have an giraph job in which the Master will read a chunk of a file from H= DFS, and then use an aggregator to broadcast the data to all vertices. No = other messages are sent, and no vertices aggregate values, only the master. In the attached logs you can see that the time spent to broadcast the data = to all vertices is slow, and seems to be hanging up somehwere. It appears = that the majority of workers receive the data in 10-15 seconds, but then no= thing happens for around 10 minutes. Log snippet shown below Is there a known reason why transmitting this data during the synchronizati= on is taking so long, or anything that can be done to speed it up? 2013-05-16 11:09:03,041 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 30 more tasks to send their aggregator= data 2013-05-16 11:09:14,444 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 10 more tasks to send their aggregator= data, task ids: [13, 20, 22, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:09:25,190 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:09:45,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:05,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:15,192 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:35,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:55,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:05,194 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:25,195 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:45,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:05,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:15,197 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:35,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:55,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:05,199 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:25,200 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:45,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:05,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:15,202 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:35,203 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:55,204 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:15,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:35,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:45,206 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:05,207 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:25,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:45,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:55,209 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:15,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:35,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:45,211 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:18:05,212 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:18:19,841 INFO org.apache.giraph.comm.netty.handler.RequestDe= coder: decode: Server window metrics MBytes/sec sent =3D 0, MBytes/sec rece= ived =3D 0.027, MBytesSent =3D 0.0006, MBytesReceived =3D 15.4028, ave sent= req MBytes =3D 0, ave received req MBytes =3D 0.0034, secs waited =3D 571.= 136 Eric Kimbrel Software Engineer I Data Fusion & Analytics Sotera Defense Solutions, Inc. o: 360-516-6621 c: 360-990-1873 e: Eric.Kimbrel@soteradefense.com w: www.potomacfusion.com | www.soteradefense.com Agility. Ingenuity. Integrity. --_000_1F592C080E9ACB4CB1C9EA1865BF3EFA0D19D534PRNMBX012TheFac_ Content-Type: text/html; charset="Windows-1252" Content-ID: <666A0599728A4A4FACD188F0BE9FBDE9@fb.com> Content-Transfer-Encoding: quoted-printable
Eric,

Can you please take a look at the logs of one of the workers listed (1= 3, 34, 38, 50, 48, 52, 58, 56), what are they doing? The fact that a worker= is waiting on aggregator can have different causes, it doesn=92t necessari= ly mean that sending aggregators is slow. It can for example mean that some workers finished computing before = others and are now waiting for others to finish and send their data. <= /div>
How big are aggregators which you are using?

Thanks,
Maja

From: Eric Kimbrel <Eric.Kimbrel@soteradefense.com>= ;
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 2:00 P= M
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Broadcast of large agg= regated value is slow.

From the attached logs in original post, you can see that both workers= use about 4 seconds of compute time on super step 4, but they complete sup= er step 4 about 10 minutes apart.


Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics

Sotera Defense Solutions, Inc.

o: 360-516-6621

c: 360-990-1873

e: Eric.Kimbrel@soteradefense.com 
w: www.potomacfusion.com |&nb= sp;www.soteradefense.com=

Agility. Ingenuity. Integrity= .



From: Eric Kimbrel <Eric.Kimbrel@soteradefense.com>= ;
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 1:50 P= M
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Broadcast of large aggrega= ted value is slow.


I have a= n giraph job in which the Master will read a chunk of a file from HDFS, and= then use an aggregator to broadcast the data to all vertices.  No oth= er messages are sent, and no vertices aggregate values, only the master.

In the a= ttached logs you can see that the time spent to broadcast the data to all v= ertices is slow, and seems to be hanging up somehwere.  It appears tha= t the majority of workers receive the data in 10-15 seconds, but then nothing happens for around 10 minutes.  Lo= g snippet shown below

Is there= a known reason why transmitting this data during the synchronization is ta= king so long, or anything that can be done to speed it up?

2013-05-16 11:09:03,041 INFO org.apache.giraph.utils.TaskIds=
PermitsBarrier: waitForRequiredPermits: Waiting for 30 more tasks to send t=
heir aggregator data
2013-05-16 11:09:14,444 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 10 more tasks to send their aggregator=
 data, task ids: [13, 20, 22, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:09:25,190 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:09:45,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:10:05,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:10:15,192 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:10:35,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:10:55,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:11:05,194 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:11:25,195 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:11:45,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:12:05,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:12:15,197 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:12:35,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:12:55,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:13:05,199 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:13:25,200 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:13:45,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:14:05,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:14:15,202 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:14:35,203 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:14:55,204 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:15:15,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:15:35,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:15:45,206 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:16:05,207 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:16:25,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:16:45,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:16:55,209 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:17:15,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:17:35,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:17:45,211 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:18:05,212 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:=
 waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator =
data, task ids: [13, 34, 38, 50, 48, 52, 58, 56]
2013-05-16 11:18:19,841 INFO org.apache.giraph.comm.netty.handler.RequestDe=
coder: decode: Server window metrics MBytes/sec sent =3D 0, MBytes/sec rece=
ived =3D 0.027, MBytesSent =3D 0.0006, MBytesReceived =3D 15.4028, ave sent=
 req MBytes =3D 0, ave received req MBytes =3D 0.0034, secs waited =3D 571.=
136






Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics

Sotera Defense Solutions, Inc.

o: 360-516-6621

c: 360-990-1873

e: Eric.Kimbrel@soteradefense.com 
w: www.potomacfusion.com |&nb= sp;www.soteradefense.com=

Agility. Ingenuity. Integrity= .


 --_000_1F592C080E9ACB4CB1C9EA1865BF3EFA0D19D534PRNMBX012TheFac_--