hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [DISCUSS/VOTE] Refactor of message queue .
Date Wed, 03 Sep 2014 09:02:40 GMT
OKay, I'll go forward.

On Fri, Aug 29, 2014 at 8:12 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> Thanks for the notice. I have no detail plan yet.
>
>
> On Friday, August 29, 2014, Chia-Hung Lin <clin4j@googlemail.com> wrote:
>>
>> Is incoming bundles manager equals to localQueueForNextIteration/
>> localQueue in 0.6.4 version?
>>
>> My understanding for localQueueForNextIteration/ localQueue is that
>> their role serves as input for the coming superstep; for instance, in
>> the N-th superstep its incoming messages are obtained from localQueue
>> (AbstractMessageManager.getCurrentMessage). In this case it look like
>> safe to save messages to local disk because once a node fails, steps
>> to recover from the previous superstep should be the same without
>> change by placing the messages saved previous to the localQueue again.
>> If not, then probably we need to consider about this issue.
>>
>>
>>
>>
>>
>>
>>
>> On 29 August 2014 08:09, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> > First of all, Our main problem is that current system requires a lot
>> > of memory space, especially graph module. As you already might know,
>> > the main memory consumer is the message queue.
>> >
>> > To solve this problem, we considered the use of local disk space e.g.,
>> > DiskQueue and SpillingQueue. However, those queues are basically not
>> > able to bundle and group the messages by destination server, in
>> > memory-efficient way. So, I don't think this approach is right way.
>> >
>> > My solution for saving the memory usage and the performance
>> > degradation, is storing serializable message objects as a byte array
>> > in queue. In graph case, 3X ~ 6X memory efficiency is expected than
>> > before (GraphJobMessage consists of destination vertex ID and message
>> > value multi-objects).
>> >
>> > In 0.6.4, Outgoing queue is replaced with outgoing bundles manager,
>> > and it showed nice memory improvement. Now I wanna start refactoring
>> > of incoming queue.
>> >
>> > My plan is that adding incoming bundles manager. Bundles can also
>> > simply be written to local disk if when memory space is not enough.
>> > So, incoming bundles manager can be performed a similar role of
>> > DiskQueue and SpillingQueue in the future.
>> >
>> > If you have any other opinion, Please let me know. If there are no
>> > objections, I'll do it.
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > CEO at DataSayer Co., Ltd.
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.



-- 
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

Mime
View raw message