giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maja Kabiljo <majakabi...@fb.com>
Subject Re: shared aggregators
Date Wed, 29 May 2013 19:16:09 GMT
Feel free to expand the explanation on the website :-)

On 5/29/13 12:00 PM, "Claudio Martella" <claudio.martella@gmail.com> wrote:

>Yes totally. Thanks.
>
>
>On Wed, May 29, 2013 at 7:19 PM, Maja Kabiljo <majakabiljo@fb.com> wrote:
>
>> Claudio,
>>
>> First, the total number of messages is the same, since worker won't be
>> sending aggregators to itself, so we have K + (N-1)*K.
>>
>> If master would be sending aggregators to all workers, it would have to
>> send K*N amount of data. This way master only sends K, and then as Greg
>> said all workers work in parallel - each sending additional K (or more
>> precisely K/N * (N-1)).
>>
>> Does this make it clearer?
>>
>> Maja
>>
>> On 5/29/13 9:22 AM, "Greg Malewicz" <malewicz@fb.com> wrote:
>>
>> >Consider parallel time.
>> >
>> >Greg
>> >
>> >On 5/29/13 9:08 AM, "Claudio Martella" <claudio.martella@gmail.com>
>> wrote:
>> >
>> >>Hi,
>> >>
>> >>I have a question about the design of shared aggregators.
>> >>Documentation says:
>> >>
>> >>"After MasterCompute.compute, master doesn't do the distribution of
>>all
>> >>aggregators to all workers, but aggregators again have their owners.
>> >>Master
>> >>only sends each aggregator to its owner, and then each worker
>>distributes
>> >>the aggregators which it owns to all other workers."
>> >>
>> >>Why are the aggregator values not sent directly by the master to all
>> >>workers, instead of doing the two hops it does now?
>> >>
>> >>Suppose I have K aggregators and N workers, the current design
>>requires K
>> >>+
>> >>K*N messages.
>> >>If the master would send the aggregators values to all the workers
>> >>directly, we would have K*N messages, or actually N messages with K
>> >>values
>> >>each.
>> >>
>> >>Am I missing something?
>> >>
>> >>Best,
>> >>Claudio
>> >>
>> >>--
>> >>   Claudio Martella
>> >>   claudio.martella@gmail.com
>> >
>>
>>
>
>
>-- 
>   Claudio Martella
>   claudio.martella@gmail.com


Mime
View raw message