incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-45) Improve the way to keep outgoing messages
Date Tue, 15 Nov 2011 21:23:51 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150788#comment-13150788
] 

Jakob Homan commented on GIRAPH-45:
-----------------------------------

A pull model, where the destination asks for messages as it needs them, or a incremental push
model (the first messages), seems like a bigger change than what I was thinking of.
bq. it can make sense for the sender to hit the disk and then send them to the other end at
the beginning of next superstep
Is this predicated on the pull/incremental-push model? My thoughts weren't based on that,
but on the current shuffle-like approach. (I'd love to have a virtual whiteboard right now.)


In a model where the outgoing messages are sent immediately, there exists the chance that
a sent message will tip the destination into having to start spilling to disk.  But that message
no longer is on the originator, so it relieves pressure on the originator and lessens the
chance the originator will need to do so. So, one worker spills and the message hits disk
once.  

If, on the other hand, we do not send outgoing messages immediately and instead spill on the
originator, when the message finally is sent, it also has the potential to tip the destination
into spilling.  If we're lucky, the destination doesn't spill and only one worker spills and
the message hits disk once.  If we're unlucky and the destination has to spill, then we have
two workers spilling and the message hitting disk twice.
                
> Improve the way to keep outgoing messages
> -----------------------------------------
>
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a potential problem
to cause out of memory when the rate of message generation is higher than the rate of message
flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or some approach
to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message