incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-45) Improve the way to keep outgoing messages
Date Mon, 28 Nov 2011 21:37:40 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158806#comment-13158806
] 

Claudio Martella commented on GIRAPH-45:
----------------------------------------

Dmitriy, could you provide an example? We're looking for disk-basked implementations that
provide the usual append-only write performance and quick scan-based read. For this reason
i thought about sequence-file based approach. I don't know any open source doing this without
doing writes in-place (tokyo-kyoto kabinet). I'm writing something like this (memstore+sequencefile+compaction)
pure java which is alpha-version now, I will speed up dev in the future, but of course it
won't be bullet-proof [https://github.com/claudiomartella/sketches]. On this aspect there's
a java open-source implementation of leveldb, but don't know if it's stable enough for us
(we could take it (or sketches) under our umbrella thought) [https://github.com/dain/leveldb].
Probably going default with something fully java (derby or something you're going to suggest)
and going pluggable is the best option.

I'll work on this from next week (moving to amsterdam next week), finishing GIRAPH-10 right
now.
                
> Improve the way to keep outgoing messages
> -----------------------------------------
>
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a potential problem
to cause out of memory when the rate of message generation is higher than the rate of message
flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or some approach
to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message