I use only one = aggregator. The code for it is extremely simple as it is not actually= used as an aggregator at all. The only use is for the master compute= class to set a large object value, and then for each vertex to read that v= alue. The Size of the Bytes writable is ~ 100M

<= span class=3D"Apple-style-span" style=3D"border-collapse: collapse;">

Heres the code for the aggregat= or:

= public class BroadcastAggregat= or extends BasicAggregator<RowList= Writable>{

// no user code ever calls aggregate.

// Master com= pute class calls setAggregatedValue once per superstep.

<= /p>

// This method is not used, but is provided for flex= ibility.

//If a non empty value is added it will be added to the = current value and then set

@Override

public void aggreg= ate(RowListWritable rw) {

if (rw.index.length =3D=3D 0) return;

if (this.getAggre= gatedValue().index.length > 0){

rw.add(this.getAggregatedValue());

}

this.setAggregatedValue(rw);

}

@Override

public RowListWritable createInitialValue() {

return new RowLis= tWritable();

}

= Eric Kimbrel
Software Engineer I Data Fusion & Analytics

o: 360-516-6621

c: 360-990-1873

e: Eric.Kimbrel@soteradefense.com=
w: www.potomacfusion.com&= nbsp;| www.soteradefense.com

= Agility. Ingenuity.&= nbsp;Integrity.

From: Maja Kabiljo &= lt;majakabiljo@fb.com>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 8:18 PM
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Broadcast of large aggregated value is slow.
Resent-From: <eric.kimbrel@soteradefense.com>

Thanks, Eric. I'm not= sure what's going on, it's strange that there are a couple of machines whi= ch wait for aggregators for a very long time, but then in exactly same mome= nt they receive them. Can you send us the code for Aggregator which you are= using? Do you know approximately how big aggregators are and how many of them do = you have?

Maja

From: Eric Kimbrel <Eric.Kimbrel@soteradefense.com>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 2:36 PM
To: "user@giraph.apache.org" <user@giraph.apache.org>
S= ubject: Re: Broadcast of large aggregated value is slow.

My apologies. Yo= u are correct, I attached the wrong long. Correct one attached here.=

Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics
Sotera Defense Solutio= ns, Inc.
o: 360-516-6621
c: 360-990-1873
e: Eric.Kimbrel@soteradefense.com
w: www.potomacfusion.com = | www.soteradefense.com
Agility. Ingenuity.= Integrity.

From: = Maja Kabiljo <majakabiljo@f= b.com>
Reply-To: "user@giraph.apache.org" &l= t;user@giraph.apache.org><= br>Date: Thursday, May 16, 2013 2:2= 5 PM
To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Broadcast of large aggregated val= ue is slow.
Resent-From: <eric.kimbrel@soteradefense.c= om>

Eric= ,

Can you please check it again, in both logs you = attached we are waiting on the worker 13 to send data, so none of those can= 't be worker 13's log.

Maja

From: Eric Kimbrel <Eric.Kimbrel@soteradefense.com>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 2:15 PM<= br>To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Broadcast of large aggregated value is = slow.

One of the attached logs is worker 13, During this time period it is= waiting for an aggregator request so that it can start the super step. &nb= sp;

Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics
Sotera Defense Solutio= ns, Inc.
o: 360-516-6621
c: 360-990-1873
e: Eric.Kimbrel@soteradefense.com
w: www.potomacfusion.com |&nb= sp;www.soteradefense.com=
Agility. = Ingenuity. Integrity
<= /div>

From: Maja= Kabiljo <majakabiljo@fb.com&g= t;
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 2:11 PM
<= span style=3D"font-weight:bold">To: "user@giraph.apache.org" <user@giraph.apache.org>
Subject: Re: Broadcast of large aggregated value is slow= .
Resent-From: <eric.kimbrel@soteradefense.com>=

Eric,

Can you please take a look at the logs of one of the worke= rs listed (13, 34, 38, 50, 48, 52, 58, 56), what are they doing? The fact t= hat a worker is waiting on aggregator can have different causes, it doesn= =92t necessarily mean that sending aggregators is slow. It can for example mean that some workers finished computing before = others and are now waiting for others to finish and send their data. <= /div>
How big are aggregators which you are using?

=
Thanks,
Maja

From: Eric Kimbrel <Eric.Kimbrel@soteradefense.com>
Reply-To: "user@giraph.apache.org" <user@giraph.apache.org>
Date: Thursday, May 16, 2013 2:00 PM
To: "user@gira= ph.apache.org" <user@= giraph.apache.org>
Subject: Re: Broadcast of large aggregated value is slow.

From the attached logs in or= iginal post, you can see that both workers use about 4 seconds of compute t= ime on super step 4, but they complete super step 4 about 10 minutes apart.=

Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics
Sotera Defense Solutio= ns, Inc.
o: 360-516-6621
c: 360-990-1873
e: Eric.Kimbrel@soteradefense.com
w: www.potomacfusion.com |&nb= sp;www.soteradefense.com=
Agility. = Ingenuity. Integrity
<= /div>

From: Eric= Kimbrel <Eric.Kimbrel= @soteradefense.com>
Reply-To: "user@giraph.apache.org= " <user@giraph.apache= .org>
Date: Thursday, May= 16, 2013 1:50 PM
To: "user@giraph.apache.org" <= user@giraph.apache.org>Subject: Broadcast of large aggreg= ated value is slow.

I have an giraph job in which the Master will r= ead a chunk of a file from HDFS, and then use an aggregator to broadcast th= e data to all vertices. No other messages are sent, and no vertices a= ggregate values, only the master.

In the attached logs you can see that the time spe= nt to broadcast the data to all vertices is slow, and seems to be hanging u= p somehwere. It appears that the majority of workers receive the data in 10-15 seconds, but then nothing happens for around 10 minutes. Lo= g snippet shown below

Is there a known reason why transmitting this data dur= ing the synchronization is taking so long, or anything that can be done to = speed it up?

2013-05-16 11:09:03,041 INFO org.apache.giraph.u= tils.TaskIdsPermitsBarrier: waitForRequiredPermits: Waiting for 30 more tas= ks to send their aggregator data 2013-05-16 11:09:14,444 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 10 more tasks to send their aggregator= data, task ids: [13, 20, 22, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:09:25,190 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:09:45,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:05,191 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:15,192 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:35,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:10:55,193 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:05,194 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:25,195 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:11:45,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:05,196 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:15,197 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:35,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:12:55,198 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:05,199 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:25,200 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:13:45,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:05,201 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:15,202 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:35,203 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:14:55,204 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:15,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:35,205 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:15:45,206 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:05,207 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:25,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:45,208 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:16:55,209 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:15,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:35,210 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:17:45,211 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:18:05,212 INFO org.apache.giraph.utils.TaskIdsPermitsBarrier:= waitForRequiredPermits: Waiting for 8 more tasks to send their aggregator = data, task ids: [13, 34, 38, 50, 48, 52, 58, 56] 2013-05-16 11:18:19,841 INFO org.apache.giraph.comm.netty.handler.RequestDe= coder: decode: Server window metrics MBytes/sec sent =3D 0, MBytes/sec rece= ived =3D 0.027, MBytesSent =3D 0.0006, MBytesReceived =3D 15.4028, ave sent= req MBytes =3D 0, ave received req MBytes =3D 0.0034, secs waited =3D 571.= 136

Eric Kimbrel
Software Engineer I&nb= sp;Data Fusion & Analytics
Sotera Defense Solutio= ns, Inc.
o: 360-516-6621
c: 360-990-1873
e: Eric.Kimbrel@soteradefense.com
w: www.potomacfusion.com |&nb= sp;www.soteradefense.com=
Agility. = Ingenuity. Integrity
<= /div>
<= /span>

--_000_CDC0FC083056EKimbrelpotomacfusioncom_--