hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Hung Lin <cli...@googlemail.com>
Subject Re: Remove Spilling Queue and rewrite checkpoint/recovery
Date Tue, 12 Aug 2014 07:11:01 GMT
I am currently working on this part based on the superstep api,
similar to the Superstep.java in the trunk.

The checkpointer[1] saves bundle message instead of single message.
Not very sure if this is what you are looking for?

[1]. https://github.com/chlin501/hama/blob/peer-comm-mech-changed/core/src/main/scala/org/apache/hama/monitor/Checkpointer.scala




On 12 August 2014 15:04, Edward J. Yoon <edwardyoon@apache.org> wrote:
> I think that transferring single messages at a time is not a wise way.
> Bundle is used to avoid network overheads and contentions. So, if we
> use Bundle, each processor always sends/receives an bundles.
>
> BSPMessageBundle is Writable (and Iterable). And it manages the
> serialized message as a byte array. If we write an bundles when
> checkpointing or using Disk-queue, it'll be more simple and faster.
>
> In Spilling Queue case, it always requires the process of unbundling
> and putting messages into queue.
>
>
> On Tue, Aug 12, 2014 at 2:41 PM, Tommaso Teofili
> <tommaso.teofili@gmail.com> wrote:
>> -1, can't we first discuss? Also it'd be helpful to be more specific on the
>> problems.
>> Tommaso
>>
>>
>>
>> 2014-08-12 4:25 GMT+02:00 Edward J. Yoon <edwardyoon@apache.org>:
>>
>>> All,
>>>
>>> I'll delete Spilling queue, and rewrite checkpoint/recovery
>>> implementation (checkpointing bundles is better than checkpointing all
>>> messages). Current implementation is quite mess :/ there are huge
>>> deserialization/serialization overheads..
>>>
>>> --
>>> Best Regards, Edward J. Yoon
>>> CEO at DataSayer Co., Ltd.
>>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> CEO at DataSayer Co., Ltd.

Mime
View raw message