hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommaso Teofili <tommaso.teof...@gmail.com>
Subject Re: [DISCUSS] Disk Queue and Spilling Queue
Date Wed, 28 May 2014 10:42:30 GMT
2014-05-28 12:21 GMT+02:00 Edward J. Yoon <edwardyoon@apache.org>:

> Even though we use the disk or spilling queue, we need to add whole
> messages into the bundle object (memory). So, I said, they are
> pointless. The bundle should be spillable. Receive-side queue is also
> the same.
>

+1


>
> If there's no objections, I'll remove the spilling queue.
>

I'm not sure that's the right move, in the latest months we've been adding
and removing things back and forth in this area.
For example beforehand we didn't bundle everything together and so the
spilling queue made sense (if I recall correctly), again I would prefer to
have everything to be configurable, so by default for example we have no
spilling queue and bundle message but we may configure things in order to
use the spilling queue and to eventually send single messages rather than
the whole bundle.

Regards,
Tommaso


>
>
> On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
> <andronat_asf@hotmail.com> wrote:
> > I am +1. That plan is going to make things much more straight to other
> developers too.
> >
> > Anastasis
> >
> > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <edwardyoon@apache.org>
> wrote:
> >
> >> Just opened https://issues.apache.org/jira/browse/HAMA-903
> >>
> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <edwardyoon@apache.org>
> wrote:
> >>> The AbstractMessageManager.loopBackMessages() method unbundle messages
> >>> into localQueueForNextIteration.  Finally, in clearOutgoingMessages()
> >>> method, localQueue is switched by localQueueForNextIteration.
> >>>
> >>> My goal is simplify this procedure. This is huge overhead.
> >>>
> >>> Basically, BSPMessageBundle provides iterator access to the messages
> >>> contained in the bundle. So, we can skip whole conversion from bundle
> >>> to queue.
> >>>
> >>>> Is it going to use rpc?
> >>>
> >>> I won't touch RPC mechanism. Only queue implementations will be
> changed.
> >>>
> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <clin4j@googlemail.com>
> wrote:
> >>>> Is it going to use rpc?
> >>>>
> >>>> Will it still use the interface, for instance, MessageManager.java?
> >>>>
> >>>> Just to check if any point for integrating with the current ongoing
> >>>> refactoring process.
> >>>>
> >>>> If possible, perhaps decoupling io part and rpc from interface would
> >>>> somehow simplify the integration progress.
> >>>>
> >>>>
> >>>> On 12 May 2014 09:01, Edward J. Yoon <edwardyoon@apache.org> wrote:
> >>>>> The old design of outgoing/incoming message queues is readable but
it
> >>>>> has some problems, and the most performance and memory issues are
> >>>>> dependent upon this part.
> >>>>>
> >>>>> 1) To send a messages to destination Peer, we serialize, compress,
> and
> >>>>> bundle the messages. So, using disk or spilling queue for the
> outgoing
> >>>>> messages is pointless and cause of degradation. This issue SOLVED
by
> >>>>> HAMA-853. We'll need to add disk-based bundle in the future.
> >>>>>
> >>>>> 2) Receive-side queue is also the same. Instead of unbundling (and
> >>>>> deserializing, decompressing) bundles into {memory, disk, or
> spilling}
> >>>>> queue, we should use bundles in efficient and asynchronous way.
> >>>>>
> >>>>> If you agree with this, I'll start to refactor the whole queue
> system.
> >>>>>
> >>>>> If you have any other ideas e.g., asynchronous message
> >>>>> synchronization, Pls let me know.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> --
> >>>>> Best Regards, Edward J. Yoon
> >>>>> CEO at DataSayer Co., Ltd.
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards, Edward J. Yoon
> >>> CEO at DataSayer Co., Ltd.
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> CEO at DataSayer Co., Ltd.
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message