incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jake Mannix (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-45) Improve the way to keep outgoing messages
Date Tue, 15 Nov 2011 20:36:53 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150751#comment-13150751
] 

Jake Mannix commented on GIRAPH-45:
-----------------------------------

So I've wanted to do this kind of thing (semi-stream the outgoing messages, instead of buffering
them all up) for practical purposes already as well.  But I've struggled with how to get it
to work properly: if the messages are so big (or so numerous) that they don't fit easily in
each mapper (together with the rest of the graph that's there), then you can help this by
sending them as you create them, but then the receiver has to buffer them on the other end...
unless the reciever is able to process them as they receive them.  

But now this is a much more complex computation model: to send out messages, you typically
need to know what you've recieved in the previous step.  But in the previous step, you've
been recieving messages and not *buffering them*, but instead *processing them* in some way.
 If this can be implemented in the Combiner, then great!  If you can't, then what option is
there except for hold them in memory somewhere, or spill to disk somewhere?
                
> Improve the way to keep outgoing messages
> -----------------------------------------
>
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a potential problem
to cause out of memory when the rate of message generation is higher than the rate of message
flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or some approach
to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message