hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andronidis Anastasios <andronat_...@hotmail.com>
Subject Re: [DISCUSS] Disk Queue and Spilling Queue
Date Tue, 13 May 2014 07:55:04 GMT
I am +1. That plan is going to make things much more straight to other developers too.

Anastasis

On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <edwardyoon@apache.org> wrote:

> Just opened https://issues.apache.org/jira/browse/HAMA-903
> 
> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> The AbstractMessageManager.loopBackMessages() method unbundle messages
>> into localQueueForNextIteration.  Finally, in clearOutgoingMessages()
>> method, localQueue is switched by localQueueForNextIteration.
>> 
>> My goal is simplify this procedure. This is huge overhead.
>> 
>> Basically, BSPMessageBundle provides iterator access to the messages
>> contained in the bundle. So, we can skip whole conversion from bundle
>> to queue.
>> 
>>> Is it going to use rpc?
>> 
>> I won't touch RPC mechanism. Only queue implementations will be changed.
>> 
>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
>>> Is it going to use rpc?
>>> 
>>> Will it still use the interface, for instance, MessageManager.java?
>>> 
>>> Just to check if any point for integrating with the current ongoing
>>> refactoring process.
>>> 
>>> If possible, perhaps decoupling io part and rpc from interface would
>>> somehow simplify the integration progress.
>>> 
>>> 
>>> On 12 May 2014 09:01, Edward J. Yoon <edwardyoon@apache.org> wrote:
>>>> The old design of outgoing/incoming message queues is readable but it
>>>> has some problems, and the most performance and memory issues are
>>>> dependent upon this part.
>>>> 
>>>> 1) To send a messages to destination Peer, we serialize, compress, and
>>>> bundle the messages. So, using disk or spilling queue for the outgoing
>>>> messages is pointless and cause of degradation. This issue SOLVED by
>>>> HAMA-853. We'll need to add disk-based bundle in the future.
>>>> 
>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
>>>> deserializing, decompressing) bundles into {memory, disk, or spilling}
>>>> queue, we should use bundles in efficient and asynchronous way.
>>>> 
>>>> If you agree with this, I'll start to refactor the whole queue system.
>>>> 
>>>> If you have any other ideas e.g., asynchronous message
>>>> synchronization, Pls let me know.
>>>> 
>>>> Thanks.
>>>> 
>>>> --
>>>> Best Regards, Edward J. Yoon
>>>> CEO at DataSayer Co., Ltd.
>> 
>> 
>> 
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
> 
> 
> 
> -- 
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.


Mime
View raw message