hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [DISCUSS] Disk Queue and Spilling Queue
Date Wed, 28 May 2014 21:45:19 GMT
Say, that's an idea. However, I think the complex option is a recipe
for confusions.

IMO, we have to finish the basics clearly. Otherwise, everything can
be fizzled out, like Fault Tolerance and Superstep API.

On Wed, May 28, 2014 at 7:42 PM, Tommaso Teofili
<tommaso.teofili@gmail.com> wrote:
> 2014-05-28 12:21 GMT+02:00 Edward J. Yoon <edwardyoon@apache.org>:
>
>> Even though we use the disk or spilling queue, we need to add whole
>> messages into the bundle object (memory). So, I said, they are
>> pointless. The bundle should be spillable. Receive-side queue is also
>> the same.
>>
>
> +1
>
>
>>
>> If there's no objections, I'll remove the spilling queue.
>>
>
> I'm not sure that's the right move, in the latest months we've been adding
> and removing things back and forth in this area.
> For example beforehand we didn't bundle everything together and so the
> spilling queue made sense (if I recall correctly), again I would prefer to
> have everything to be configurable, so by default for example we have no
> spilling queue and bundle message but we may configure things in order to
> use the spilling queue and to eventually send single messages rather than
> the whole bundle.
>
> Regards,
> Tommaso
>
>
>>
>>
>> On Tue, May 13, 2014 at 4:55 PM, Andronidis Anastasios
>> <andronat_asf@hotmail.com> wrote:
>> > I am +1. That plan is going to make things much more straight to other
>> developers too.
>> >
>> > Anastasis
>> >
>> > On 13 Μαϊ 2014, at 7:27 π.μ., Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>> >
>> >> Just opened https://issues.apache.org/jira/browse/HAMA-903
>> >>
>> >> On Tue, May 13, 2014 at 10:51 AM, Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>> >>> The AbstractMessageManager.loopBackMessages() method unbundle messages
>> >>> into localQueueForNextIteration.  Finally, in clearOutgoingMessages()
>> >>> method, localQueue is switched by localQueueForNextIteration.
>> >>>
>> >>> My goal is simplify this procedure. This is huge overhead.
>> >>>
>> >>> Basically, BSPMessageBundle provides iterator access to the messages
>> >>> contained in the bundle. So, we can skip whole conversion from bundle
>> >>> to queue.
>> >>>
>> >>>> Is it going to use rpc?
>> >>>
>> >>> I won't touch RPC mechanism. Only queue implementations will be
>> changed.
>> >>>
>> >>> On Mon, May 12, 2014 at 11:59 PM, Chia-Hung Lin <clin4j@googlemail.com>
>> wrote:
>> >>>> Is it going to use rpc?
>> >>>>
>> >>>> Will it still use the interface, for instance, MessageManager.java?
>> >>>>
>> >>>> Just to check if any point for integrating with the current ongoing
>> >>>> refactoring process.
>> >>>>
>> >>>> If possible, perhaps decoupling io part and rpc from interface would
>> >>>> somehow simplify the integration progress.
>> >>>>
>> >>>>
>> >>>> On 12 May 2014 09:01, Edward J. Yoon <edwardyoon@apache.org>
wrote:
>> >>>>> The old design of outgoing/incoming message queues is readable
but it
>> >>>>> has some problems, and the most performance and memory issues
are
>> >>>>> dependent upon this part.
>> >>>>>
>> >>>>> 1) To send a messages to destination Peer, we serialize, compress,
>> and
>> >>>>> bundle the messages. So, using disk or spilling queue for the
>> outgoing
>> >>>>> messages is pointless and cause of degradation. This issue SOLVED
by
>> >>>>> HAMA-853. We'll need to add disk-based bundle in the future.
>> >>>>>
>> >>>>> 2) Receive-side queue is also the same. Instead of unbundling
(and
>> >>>>> deserializing, decompressing) bundles into {memory, disk, or
>> spilling}
>> >>>>> queue, we should use bundles in efficient and asynchronous way.
>> >>>>>
>> >>>>> If you agree with this, I'll start to refactor the whole queue
>> system.
>> >>>>>
>> >>>>> If you have any other ideas e.g., asynchronous message
>> >>>>> synchronization, Pls let me know.
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> --
>> >>>>> Best Regards, Edward J. Yoon
>> >>>>> CEO at DataSayer Co., Ltd.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Best Regards, Edward J. Yoon
>> >>> CEO at DataSayer Co., Ltd.
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> CEO at DataSayer Co., Ltd.
>> >
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> CEO at DataSayer Co., Ltd.
>>



-- 
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

Mime
View raw message