incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dmit...@twitter.com>
Subject Re: Message processing
Date Thu, 08 Sep 2011 21:57:33 GMT
At this point, -dev and -user is probably pretty much the same set of people :)

On Thu, Sep 8, 2011 at 2:33 PM, Avery Ching <aching@apache.org> wrote:
> On 9/8/11 2:19 PM, Jake Mannix wrote:
>
> On Thu, Sep 8, 2011 at 2:01 PM, Avery Ching <aching@apache.org> wrote:
>>
>> Yes, the messages are delivered to inMessages (map of id -> list of
>> messages).  Then they are transferred to the vertices just before the
>> computation on a worker.
>
> So just to make sure I understand correctly (I think I do, but reasoning
> about distributed async operations makes my head hurt a bit), I'm going to
> sound dumb and repeat this back to you: they show up in an asynchronous
> manner into BasicRPCCommunications, where they get buffered up into the
> inMessages map (one list for each vertex on the local node), where the
> GraphMapper then loops through and gives them to each Vertex before they do
> their compute() calls?
>
>
> You have got it =).  Although to clarify, all the messages are attached to
> the Vertex in the commService.prepareSuperstep() in GraphMapper.  I'm
> starting to wonder if we should have this conversation on
> giraph-dev@incubator.apache.org =).  Never tried that list yet though...
>
>
>> On 9/8/11 1:55 PM, Jake Mannix wrote:
>>
>> So in particular, each GraphMapper has a BasicRPCCommunications object
>> (and a CommunicationsInterface proxy object for each of the other nodes),
>> and when other nodes call putMsg(vertexId, message) on their respective
>> proxies, it shows up in the target vertex's BasicRPCCommunions putMsg()
>> call, is that right?
>>
>> On Thu, Sep 8, 2011 at 1:38 PM, Avery Ching <aching@apache.org> wrote:
>>>
>>> If I'm understanding your question correctly, the iterator is created in
>>> GraphMapper, just before compute() is called.  Messages are tranferred to
>>> the vertices prior to the compute portion superstep.
>>>
>>> Avery
>>>
>>> On 9/8/11 12:59 PM, Jake Mannix wrote:
>>>>
>>>> Question about the message passing API:
>>>>
>>>>  BaseVertex#sendMsg(I id, M msg)
>>>>
>>>> sends messages.  And
>>>>
>>>>  BaseVertex#compute(Iterator<M> msgIterator)
>>>>
>>>> deals with them in a big iterator provided by the framework.  My
>>>> question is where the Iterator of messages is created?  If I wanted to say,
>>>> subvert the processing to avoid buffering them up into one big List, and
>>>> instead handle them as they come in, where would I look in the code to see
>>>> where it's buffered up?
>>>>
>>>>  -jake
>>>>
>>>>
>>>
>>
>>
>
>
>



-- 
Dmitriy V Ryaboy
Twitter Analytics
http://twitter.com/squarecog

Mime
View raw message