giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hassan Eslami (JIRA)" <j...@apache.org>
Subject [jira] [Created] (GIRAPH-1025) Flow-control mechanism improving out-of-core feature
Date Sun, 02 Aug 2015 05:12:04 GMT
Hassan Eslami created GIRAPH-1025:
-------------------------------------

             Summary: Flow-control mechanism improving out-of-core feature
                 Key: GIRAPH-1025
                 URL: https://issues.apache.org/jira/browse/GIRAPH-1025
             Project: Giraph
          Issue Type: Improvement
          Components: graph
            Reporter: Hassan Eslami
            Assignee: Hassan Eslami


Current implementation of out-of-core mechanism may fail due to message flooding in applications
without message combiner. In such applications the rate at which messages arrive may be much
more than the rate of read/write to disk in OOC mechanism. In this scenario, the OOC mechanism
will get OOM as it cannot keep up with storing incoming messages in-memory.

One might say this problem may also appear in input-superstep as we have input vertices and
edges going around as messages, but note that in input superstep the rate at which a worker
receive messages (vertices and edges) are almost the same as the rate of reading from disk,
and that is almost the same as the rate at which offloading data to disk happens. However,
this flow-control mechanism can be generalized for input superstep too. At this point, this
generalization does not seem necessary though.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message