hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HAMA-554) Implement a new message manager which writes messages to HDFS rather than sending them over RPC.
Date Mon, 10 Mar 2014 02:26:43 GMT

     [ https://issues.apache.org/jira/browse/HAMA-554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Edward J. Yoon resolved HAMA-554.
---------------------------------

    Resolution: Won't Fix

I don't think this is good idea. I'd rather use local fs and scp than hdfs.

> Implement a new message manager which writes messages to HDFS rather than sending them
over RPC.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HAMA-554
>                 URL: https://issues.apache.org/jira/browse/HAMA-554
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp core, messaging
>            Reporter: Apurv Verma
>
> The current message managers send all messages over RPC. In this issue we would like
to create a DFSMessageManager which writes all messages to HDFS and store its PATH. Later
in the sync() phase these PATHs may be send to other peers which can then read the messages
from the HDFS.
> Pros:
> This can handle the case when the number of messages sent in a single superstep are too
many to be all contained in memory. HAMA-521 already provides the implementation of Disk based
queue. We can abstract this DiskQueue in the util section and use it here.
> Cons:
> Reading data from HDFS might be slow.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message