hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [DISCUSS] Disk Queue and Spilling Queue
Date Tue, 13 May 2014 01:51:57 GMT
The AbstractMessageManager.loopBackMessages() method unbundle messages
into localQueueForNextIteration.  Finally, in clearOutgoingMessages()
method, localQueue is switched by localQueueForNextIteration.

My goal is simplify this procedure. This is huge overhead.

Basically, BSPMessageBundle provides iterator access to the messages
contained in the bundle. So, we can skip whole conversion from bundle
to queue.

> Is it going to use rpc?

I won't touch RPC mechanism. Only queue implementations will be changed.

On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
> Is it going to use rpc?
> Will it still use the interface, for instance, MessageManager.java?
> Just to check if any point for integrating with the current ongoing
> refactoring process.
> If possible, perhaps decoupling io part and rpc from interface would
> somehow simplify the integration progress.
> On 12 May 2014 09:01, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> The old design of outgoing/incoming message queues is readable but it
>> has some problems, and the most performance and memory issues are
>> dependent upon this part.
>> 1) To send a messages to destination Peer, we serialize, compress, and
>> bundle the messages. So, using disk or spilling queue for the outgoing
>> messages is pointless and cause of degradation. This issue SOLVED by
>> HAMA-853. We'll need to add disk-based bundle in the future.
>> 2) Receive-side queue is also the same. Instead of unbundling (and
>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>> queue, we should use bundles in efficient and asynchronous way.
>> If you agree with this, I'll start to refactor the whole queue system.
>> If you have any other ideas e.g., asynchronous message
>> synchronization, Pls let me know.
>> Thanks.
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.

Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

View raw message