hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Hung Lin <cli...@googlemail.com>
Subject Re: [DISCUSS] Disk Queue and Spilling Queue
Date Mon, 12 May 2014 14:59:52 GMT
Is it going to use rpc?

Will it still use the interface, for instance, MessageManager.java?

Just to check if any point for integrating with the current ongoing
refactoring process.

If possible, perhaps decoupling io part and rpc from interface would
somehow simplify the integration progress.


On 12 May 2014 09:01, Edward J. Yoon <edwardyoon@apache.org> wrote:
> The old design of outgoing/incoming message queues is readable but it
> has some problems, and the most performance and memory issues are
> dependent upon this part.
>
> 1) To send a messages to destination Peer, we serialize, compress, and
> bundle the messages. So, using disk or spilling queue for the outgoing
> messages is pointless and cause of degradation. This issue SOLVED by
> HAMA-853. We'll need to add disk-based bundle in the future.
>
> 2) Receive-side queue is also the same. Instead of unbundling (and
> deserializing, decompressing) bundles into {memory, disk, or spilling}
> queue, we should use bundles in efficient and asynchronous way.
>
> If you agree with this, I'll start to refactor the whole queue system.
>
> If you have any other ideas e.g., asynchronous message
> synchronization, Pls let me know.
>
> Thanks.
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.

Mime
View raw message