hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: outgoingQueue and messages bundling
Date Wed, 15 Jan 2014 08:44:28 GMT
If we use Spilling or Disk queues for outgoing queue, some memory
consumption can be saved but we can't avoid deserializing and bundling
messages overhead. According to my test, disk-based outgoing queue
doesn't help much in memory usage.

So, in my opinion, we should add outgoing messages in bundle object
directly, and should consider about byte array bundle, or disk-based
bundle. We can add something like OutgoingMessageManager.

Does it make sense?

On Wed, Jan 15, 2014 at 2:32 PM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> As I described here,
> http://wiki.apache.org/hama/Architecture#Communication_and_Synchronization_Process
> Currently, we puts the messages to be sent into outgoing queue. Then
> we puts the messages of outgoing queue into BSPMessageBundle object
> again.
> This is huge overhead, and very inefficient in memory usage. Why
> should we do like that?
> Please let me know if i missing something important.
> --
> Best Regards, Edward J. Yoon
> @eddieyoon

Best Regards, Edward J. Yoon

View raw message