hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MaoYuan Xian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-766) Implements network based send queue
Date Thu, 20 Jun 2013 10:13:20 GMT

    [ https://issues.apache.org/jira/browse/HAMA-766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689101#comment-13689101

MaoYuan Xian commented on HAMA-766:

Yes, I agree with Suraj, we need spilled data processor -> Write to file + send to network.
In current mechanism, message dispatcher (send to network) does not start until all data prepared
(message sending starts in BSPPeerImpl.sync()). 
Maybe spilling queue will implement the network sending concurrently with local file writing
(message dispatch does not need to wait for disk operations), mark this issue as duplicated
with SpillingQueue maybe? Sorry for unclear about the spillingQueue developing road map.

I did a very beginning implementations (no combine and file saving) in https://github.com/kennethxian/hama/commit/0effb82ae19f3bde059d2d94169824a2c5e1b046.
But, some message lost when handle big data. Because of the trick implementations in DirectByteBufferInputStream,
I still can not find out the root cause (Problem maybe in the sepearted one messeg into two
buffer or de-serialize them). 
However, I think, for distributing message to network, we can skip the serialize/deserialize
process between Queue add and message bundle sending, right?
> Implements network based send queue
> -----------------------------------
>                 Key: HAMA-766
>                 URL: https://issues.apache.org/jira/browse/HAMA-766
>             Project: Hama
>          Issue Type: Improvement
>          Components: bsp core
>            Reporter: MaoYuan Xian
>            Priority: Minor
> I would like to implement a so call directly send queue for performance enhancement.
Currently, we have the memory queue, DiskQueue and SpillingQueue. The memory queue need to
hold all of the message before send them out, DiskQueue and SpillingQueue need to operate
Disk and cause some performance reduction.
> Can we implement a send queue? Something like SpillingQueue, the difference is, it will
spill message directly to network instead of to disk. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message