hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [DISCUSS/VOTE] Refactor of message queue .
Date Fri, 29 Aug 2014 11:12:19 GMT
Thanks for the notice. I have no detail plan yet.

On Friday, August 29, 2014, Chia-Hung Lin <clin4j@googlemail.com> wrote:

> Is incoming bundles manager equals to localQueueForNextIteration/
> localQueue in 0.6.4 version?
>
> My understanding for localQueueForNextIteration/ localQueue is that
> their role serves as input for the coming superstep; for instance, in
> the N-th superstep its incoming messages are obtained from localQueue
> (AbstractMessageManager.getCurrentMessage). In this case it look like
> safe to save messages to local disk because once a node fails, steps
> to recover from the previous superstep should be the same without
> change by placing the messages saved previous to the localQueue again.
> If not, then probably we need to consider about this issue.
>
>
>
>
>
>
>
> On 29 August 2014 08:09, Edward J. Yoon <edwardyoon@apache.org
> <javascript:;>> wrote:
> > First of all, Our main problem is that current system requires a lot
> > of memory space, especially graph module. As you already might know,
> > the main memory consumer is the message queue.
> >
> > To solve this problem, we considered the use of local disk space e.g.,
> > DiskQueue and SpillingQueue. However, those queues are basically not
> > able to bundle and group the messages by destination server, in
> > memory-efficient way. So, I don't think this approach is right way.
> >
> > My solution for saving the memory usage and the performance
> > degradation, is storing serializable message objects as a byte array
> > in queue. In graph case, 3X ~ 6X memory efficiency is expected than
> > before (GraphJobMessage consists of destination vertex ID and message
> > value multi-objects).
> >
> > In 0.6.4, Outgoing queue is replaced with outgoing bundles manager,
> > and it showed nice memory improvement. Now I wanna start refactoring
> > of incoming queue.
> >
> > My plan is that adding incoming bundles manager. Bundles can also
> > simply be written to local disk if when memory space is not enough.
> > So, incoming bundles manager can be performed a similar role of
> > DiskQueue and SpillingQueue in the future.
> >
> > If you have any other opinion, Please let me know. If there are no
> > objections, I'll do it.
> >
> > --
> > Best Regards, Edward J. Yoon
> > CEO at DataSayer Co., Ltd.
>


-- 
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message