incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Message processing
Date Thu, 08 Sep 2011 21:33:37 GMT
On 9/8/11 2:19 PM, Jake Mannix wrote:
>
>
> On Thu, Sep 8, 2011 at 2:01 PM, Avery Ching <aching@apache.org 
> <mailto:aching@apache.org>> wrote:
>
>     Yes, the messages are delivered to inMessages (map of id -> list
>     of messages).  Then they are transferred to the vertices just
>     before the computation on a worker.
>
>
> So just to make sure I understand correctly (I think I do, but 
> reasoning about distributed async operations makes my head hurt a 
> bit), I'm going to sound dumb and repeat this back to you: they show 
> up in an asynchronous manner into BasicRPCCommunications, where they 
> get buffered up into the inMessages map (one list for each vertex on 
> the local node), where the GraphMapper then loops through and gives 
> them to each Vertex before they do their compute() calls?
You have got it =).  Although to clarify, all the messages are attached 
to the Vertex in the commService.prepareSuperstep() in GraphMapper.  I'm 
starting to wonder if we should have this conversation on 
giraph-dev@incubator.apache.org =).  Never tried that list yet though...

>
>     On 9/8/11 1:55 PM, Jake Mannix wrote:
>>     So in particular, each GraphMapper has a BasicRPCCommunications
>>     object (and a CommunicationsInterface proxy object for each of
>>     the other nodes), and when other nodes call putMsg(vertexId,
>>     message) on their respective proxies, it shows up in the target
>>     vertex's BasicRPCCommunions putMsg() call, is that right?
>>
>>     On Thu, Sep 8, 2011 at 1:38 PM, Avery Ching <aching@apache.org
>>     <mailto:aching@apache.org>> wrote:
>>
>>         If I'm understanding your question correctly, the iterator is
>>         created in GraphMapper, just before compute() is called.
>>          Messages are tranferred to the vertices prior to the compute
>>         portion superstep.
>>
>>         Avery
>>
>>
>>         On 9/8/11 12:59 PM, Jake Mannix wrote:
>>
>>             Question about the message passing API:
>>
>>              BaseVertex#sendMsg(I id, M msg)
>>
>>             sends messages.  And
>>
>>              BaseVertex#compute(Iterator<M> msgIterator)
>>
>>             deals with them in a big iterator provided by the
>>             framework.  My question is where the Iterator of messages
>>             is created?  If I wanted to say, subvert the processing
>>             to avoid buffering them up into one big List, and instead
>>             handle them as they come in, where would I look in the
>>             code to see where it's buffered up?
>>
>>              -jake
>>
>>
>>
>>
>
>


Mime
View raw message