hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suraj Menon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-766) Implements network based send queue
Date Thu, 20 Jun 2013 02:43:21 GMT

    [ https://issues.apache.org/jira/browse/HAMA-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688809#comment-13688809
] 

Suraj Menon commented on HAMA-766:
----------------------------------

We may need the messages to be still written to disk for a complete fault tolerance solution.
The spilling queue was designed with async network communication in mind. The WriteSpilledDataProcessor
can have a handle to say MessageDispatcher class that dispatches the SpilledByteBuffer as
a ByteBufferMessageBundle to the peer. The trick is whether the spilling thread would block
the peer thread. We can make the message dispatcher to read from the files written by spilling
queues and send them asynchronously. We can evaluate both approaches. Let me know what your
take is. The spilled data processor gets pipelined as Combiner spilled data processor ->
Write to file processor + Send on n/w processor. Thus we can get the combiner working on spilled
data.
                
> Implements network based send queue
> -----------------------------------
>
>                 Key: HAMA-766
>                 URL: https://issues.apache.org/jira/browse/HAMA-766
>             Project: Hama
>          Issue Type: Improvement
>          Components: bsp core
>            Reporter: MaoYuan Xian
>            Priority: Minor
>
> I would like to implement a so call directly send queue for performance enhancement.
Currently, we have the memory queue, DiskQueue and SpillingQueue. The memory queue need to
hold all of the message before send them out, DiskQueue and SpillingQueue need to operate
Disk and cause some performance reduction.
> Can we implement a send queue? Something like SpillingQueue, the difference is, it will
spill message directly to network instead of to disk. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message