incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-45) Improve the way to keep outgoing messages
Date Fri, 16 Dec 2011 22:30:31 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171273#comment-13171273
] 

Avery Ching commented on GIRAPH-45:
-----------------------------------

Ah, I get it.  You always dump the whole map of sorted vertices and messages lists to a file
with a BTree index and BloomFilter.  Then on read, you only load the initial parts of each
files (BTree and BlooomFilter) to see if it has that vertex.

So in conclusion:

Partition-based out-of-core:
+ Only one seek per partition
- Must be able to keep all messages from any partition in memory at a given time, or use the
recursive split solution

Sorted-map out-of-core:
+ Loads the minimum number of messages at a time into memory
- In the worst case, the number of seeks per vertex could be equal to the number of flushes
of the sorted memory map

Does this sound about right?

If I'm correct in my understanding of your idea, then does the SortedMap<I, Queue<M>>
concurentskiplistmap need to be sorted?  Couldn't a HashMap work?  I'm not sure what the sorting
buys you.
                
> Improve the way to keep outgoing messages
> -----------------------------------------
>
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a potential problem
to cause out of memory when the rate of message generation is higher than the rate of message
flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or some approach
to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message