incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HAMA-554) Implement a new message manager which writes messages to HDFS rather than sending them over RPC.
Date Mon, 16 Apr 2012 16:52:17 GMT

     [ https://issues.apache.org/jira/browse/HAMA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Jungblut updated HAMA-554:
---------------------------------

    Component/s: messaging
                 bsp
         Labels:   (was: features)
    
> Implement a new message manager which writes messages to HDFS rather than sending them
over RPC.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HAMA-554
>                 URL: https://issues.apache.org/jira/browse/HAMA-554
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp, messaging
>            Reporter: Apurv Verma
>
> The current message managers send all messages over RPC. In this issue we would like
to create a DFSMessageManager which writes all messages to HDFS and store its PATH. Later
in the sync() phase these PATHs may be send to other peers which can then read the messages
from the HDFS.
> Pros:
> This can handle the case when the number of messages sent in a single superstep are too
many to be all contained in memory. HAMA-521 already provides the implementation of Disk based
queue. We can abstract this DiskQueue in the util section and use it here.
> Cons:
> Reading data from HDFS might be slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message